date,title,content,url
2022-08-16T13:59:12.000Z,How Density Manages Large Real Estate Portfolios Using TimescaleDB,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, part of the team atDensity—Shane Steidley, director of software, andBrock Friedrich, software engineer—explain how they are using TimescaleDB to query data from IoT devices to help companies with substantial real estate footprints make data-driven decisions to optimize their real estate usage while reducing carbon emissions.About the CompanyShane:Density builds technology that helps companies understand how people use physical space. Founded in 2014, our platform provides workplace analytics enabled by custom-built sensors which capture the complexities of how people move through and use space without invading privacy. These sensors feed raw data to Density’s analytics platform—a fast, data-rich system that provides comprehensive insights into how spaces are used, allowing companies to easily compare a space's performance against its intended purpose, another space, or portfolio benchmarks.The platform can reveal if employees are choosing focus space or collaborative space, which floors are the most popular (and why), and how one office’s occupancy rate compares to others in a portfolio or against industry benchmarks. Today, our technology is helping inform decisions for the workplaces of some of the largest companies in the world, spanning 32 countries, with more than 1.25 billion square feet under management. Our growing list of customers ranges from Fortune 100 to high-growth tech companies that see the value in understanding the performance of their office spaces to lower operational costs, improve employee experience, and reduce their carbon footprint.Our ultimate mission is to measure and improve humanity’s footprint on the world—optimizing commercial real estate portfolios is just the first step. Thirty-nine percent of CO2  emissions are directly or indirectly linked to real estate construction, so if we can help companies measure and optimize their use of real estate, we can have an outsized impact on the different issues that are impacting climate.A tweet from the Density Team on Earth DayWith Density, customers can understand how their space is used and right-size it. When we started, we needed to sell the problem and then sell our solution. Post-COVID, everybody understands the problem and the data that they could have access to, and how Density can provide value.Brock: Density also provides that value while maintaining the anonymity of each person that walks underneath our sensors.Shane: Privacy is very important to us; it’s one of our core tenets. And because of this, we don’t use cameras—we use other sensor types, namely infrared and radar, that don’t capture any personally identifiable information (PII).About the TeamShane: Density was born out of a web development consultancy. The consultancy founders got tired of walking to their favorite coffee shop during the cold, harsh winter in upstate New York only to find a long line. This unpleasant experience gave them the idea to create an occupancy sensor.As you might expect, given the company’s origin, the Density engineering team started with 4-5 engineers with a heavy web development background. We have since grown our engineering team into a team capable of performing what I call true full-stack development. We have an electrical engineering team that designs our hardware and lays out the boards, and our devices are manufactured in Syracuse, NY, where we also have an R&D lab. Our team includes experts in mechanical design and embedded software, and we now have a backend system with multiple pipelines (whose data typically ends up in TimescaleDB) and multiple React-based frontend applications.Brock: As a team, there are many different daily routines. It’s a mix of running production systems like Shane mentioned, live data pipelines, running historical analysis, or writing new features.Shane:And we’re hiring! If you are interested in joining the Density Team,please check out our jobs page!About the ProjectShane: We use different types of data at different stages. Our entry sensors generate infrared data that is very dense. It’s not feasible to send this dense data to the cloud,  so we have to do all processing and machine learning at the edge (on the device).  The results of this edge processing are +1/-1 counts that can be aggregated in our pipelines and TimescaleDB.Our radar-based sensors generate sparse data, so we do less processing at the edge and more processing in the backend. That data has to go through a pipeline and get transformed before it makes sense to insert it into TimescaleDB and perform any aggregations. TimescaleDB really provides value when it comes to querying the data, allowing customers to slice the data up in multiple dimensions. That is something that just wasn’t easy before we started using TimescaleDB.Brock:Yeah, to tack onto that, in TimescaleDB we store counts of people in spaces over time, and a wealth of derivative metrics off of those people counts, things like how long were people in spaces, not specific people, but how long was what we call dwell time?How long was this space used continuously, or what was its usage compared to similar space types throughout the day?One of the parts I think Shane was trying to highlight is that there's a real dynamism to how those queries can take shape in that they can be sliced and diced and composed in a more or less arbitrary number of ways. And TimescaleDB’s flexibility—it is built on a relational model at the end of the day—and its ability to do the partitioning under the covers and thehypertablesto let us access all of the data back in time very quickly is the magic combination that we were looking for in a time-series database to meet our use case.Choosing (and Using!) TimescaleDBBrock: I found out about Timescale back in a previous life, circa early 2019. I worked for an oil and gas firm and was doing a lot of research into time-series storage and the options available in that space because we were developing a data software solution for supervisory control and data acquisition. Essentially, it boiled down to real-time remote sensing and monitoring for industrial controls.During that research, I happened across TimescaleDB, which was still pretty early on. It was right about the timecontinuous aggregatescame out, which was one of the big selling points for me. So when I came to Density, they were just beginning to evaluate options for time-series databases for our applications. I was able to contribute my previous experience with TimescaleDB to that process. As we evaluated the options, TimescaleDB came out as the clear winner, and the rest is history.✨Editor’s Note:Read our documentationto learn more about continuous aggregates.Shane:As an IoT company, we’ve had sensors since the very beginning. And when we started, a lot of the engineering staff came from a web consultancy, so I don’t think we did realize from the beginning thatwe needed a time-series database or even quite knew what a time-series database was.“I think moving forward, TimescaleDB, at least in my opinion, is just going to be the default time-series database”Shane SteidleyWhat seems obvious to us now wasn’t obvious back in 2017, when we built an entire time-series database using stored procedures and vanilla PostgreSQL. It was pretty cool when we finally brought over TimescaleDB. We were like: “Oh, it just does all this for us! Look, there’s a bucket function, and it’s going to return the right size buckets that you need. And it handles all of the weirdness of time zones and daylight savings time.” And you can ingest all this data, whereas before, because of how we were using PostgreSQL, we would struggle with ingesting the amount of data we needed to ingest.I think moving forward, TimescaleDB, at least in my opinion, is just going to be the default time-series database. I think you're just going to have to have a reason not to use TimescaleDB because it's so simple and fits in with PostgreSQL.Brock:The top factors that led us to TimescaleDB specifically were its tolerance for high insert rates. It's just blown away all of our expectations, even based on benchmarks that we were able to see online at the time. It's built on top of PostgreSQL, as Shane talked about earlier. There's very little in the way of a special TimescaleDB domain-specific language, and it's operationally very familiar to operating vanilla PostgreSQL.Both of those were huge wins, just both operationally and development-wise. We always have the ability to fall back on core PostgreSQL principles or relational data models as we need to, but we also have the capability to dive deeper into TimescaleDB’s specific functionality to meet those big time-series use cases.“I get the most pride in doing plain SQL against TimescaleDB, getting time-series results at scale, and not having to do a bunch of backflips”Brock FriedrichShane:We use TimescaleDB like a time-series database should be used. We use it not just for the continuous aggregates of count data and other metrics that Brock's touched on, but the bucketing, the things that are so complicated if you push them to application code. When handled in TimescaleDB, it just gives you the right data the way that you want it. There are obviously some edge cases, but 99 % of the time, TimescaleDB just does what you want it to do.Brock: What would we use if TimescaleDB didn't exist is one of those topics that I like not to think about because it gives me anxiety. There are plenty of other time-series database options, but none fit the cross-section of requirements that at least Density has in quite the manner that TimescaleDB does. So whenever somebody else asks me that question, like in passing, I just say, “Let's just pretend like that's not possible and go about our business.”I get really proud of querying against TimescaleDB whenever we run really simple queries, like selecting the data from one of our sensors for a day from six years ago, and we run that at scale, and it runs like it would run in a much smaller scale like we only had a few months of data. And that goes back to one of the things I was appreciating earlier, which is that chunk exclusion –– that ability to operate what would be these really large query plans for vanilla PostgreSQL, and trim them down to something that’s predictable and reasonable, and operates at relatively low latencies. So all that’s to say, I get the most pride in doing plain SQL against TimescaleDB, getting time-series results at scale, and not having to do a bunch of backflips./* This query yields space count aggregates at 10-minute resolution for a given set of spaces over a three-day period,
   where each 10-minute bucket is represented in the output, even if that bucket contains no data.The query first
   selects data from a sparse one-minute cagg and unions that data to a set of empty records, generated with the Postgres
   generate_series function, then rolls up the unioned records into 10-minute aggregates.
   The union against the set of empty records ensures that all 10-minute intervals are represented in the final results.
   This step is necessary as the one-minute data is sparse, meaning a given 10-minute interval could contain no data, and
   the time_bucket_gapfill function does not register that a bucket needs to be injected if no records exist within
   an interval.
 */

select und.space_id,
       time_bucket('10m', und.inner_bucket)                                                        as bucket,
       min(und.occupancy_min)                                                                      as occupancy_min,
       max(und.occupancy_max)                                                                      as occupancy_max,
       first(und.first_occupancy, und.inner_bucket) filter (where und.first_occupancy is not null) as first_occupancy,
       last(und.last_occupancy, und.inner_bucket) filter (where und.last_occupancy is not null)    as last_occupancy
from (select c1m.bucket          as inner_bucket,
             c1m.space_id        as space_id,
             c1m.occupancy_min   as occupancy_min,
             c1m.occupancy_max   as occupancy_max,
             c1m.first_occupancy as first_occupancy,
             c1m.last_occupancy  as last_occupancy
      from cav_space_counts_1m c1m
      where c1m.bucket between '2022-05-22 13:00:00+0000' and '2022-05-25 13:00:00+0000'
        and c1m.space_id in (997969122178368367, 997969123637986180)
      union
      select time_bucket_gapfill('10m', generate_series,
                                 '2022-05-22 13:00:00+0000',
                                 '2022-05-25 13:00:00+0000') as inner_bucket,
             space_id,
             null                                            as occupancy_min,
             null                                            as occupancy_max,
             null                                            as first_occupancy,
             null                                            as last_occupancy
      from generate_series('2022-05-22 13:00:00+0000'::timestamptz,
                           '2022-05-25 13:00:00+0000'::timestamptz,
                           '10m')
               join unnest(array [997969122178368367, 997969123637986180]) as space_id on true) as und
group by und.space_id, bucket
order by bucket;✨Editor’s Note:Learn more about chunk exclusionin this blog post about how we fixed long-running now ( ) queries (and made them lightning fast), orread our docs for more info on hypertables and chunks.Current Deployment and Future PlansBrock:For data visualization, we useTableauandGrafana.The primary languages we use that interact with TimescaleDB are Rust and Python. Bonus: big shout out to JetBrains for their database IDE,DataGrip. It is the best on the market by a wide margin.✨Editor’s Note:Slow Grafana performance?Learn how to fix using downsampling in one of our recent blog posts.We find TimescaleDB to be very simple to use, just flat-out, dead simple. Any TimescaleDB-specific semantics, all of the defaults always take you a long way toward meeting whatever use case you're setting out to achieve. The narrative and API documentation online is first class, in my opinion.But maybe the most telling point is that there’s very little shock value whenever you’re discovering new feature value or features within TimescaleDB. And by shock value, I mean whenever I discover something like a continuous aggregate, for example, it operates conceptually almost identically to how vanilla PostgreSQL materialized view operates, but there’s extra pizazz on top that TimescaleDB does to meet the real-time component of the materialized view, refresh in the background, and all that.So I don't really want to undersell whatTimescaleDB is doing under the covers to make the magic happen. But, from an end perspective, coming from a PostgreSQL background, many of the features align conceptually with what I would expect if I was just writing something against the vanilla PostgreSQL.RoadmapShane:When we started, we threw everything at TimescaleDB, and now we’re being more responsible users of TimescaleDB. We’re at that sweet spot within TimescaleDB where it’s found product-market fit within Density.Brock:It’s hard to name my favorite TimescaleDB feature, but continuous aggregates have been a game-changer in a variety of different ways. Early on, whenever we first onboarded to TimescaleDB and deployed, it made development significantly faster, and we could just offload a lot of decision logic and complexity around time zone handling and bucketing (that’s a big one) to TimescaleDB and let it handle that complexity for us.Continuous aggregates come into play in that we were able to roll up multiple resolutions of our sensor account data, our people count data, and make that available in a much more efficient way with little-to-no effort so far as the code that we actually had to write to deliver those efficiencies.Continuous aggregates are becoming less important for us now than they were in our previous usage, just because as Shane was talking about earlier on, we're growing to such a size, and our use cases are becoming so complex that we're wanting to move to more the ETL (extract, transform, load) type of processing out of the database. So we're not monopolizing database resources to do some of those computations and move them upstream of the database into processing pipelines and still take advantage of continuous aggregates but to a lesser degree. We're doing less of the mathematical stuff in the database, but we're still using continuous aggregates to roll up high-resolution data to lower resolutions.Advice & ResourcesBrock:If I had to recommend resources, the first would probably be theTimescale blog. It’s been historically pretty informative over the years about the internals, like what’s happening in TimescaleDB underpinning a specific feature. One that I remember specifically is explaining thevarious compression algorithms in play for different data types within PostgreSQL. Being able to distill that knowledge down to a blog article that I could consume and then assimilate into my mental model of what was going on under the covers of the technology I was using has been helpful repeatedly.The advice that I would give for building a scalable database or a strategy around that is that when designing for an analytic workload specifically, don't direct any read load to the master or in standby notes. Always use a read replica for enough reasons that we probably don't have enough time to talk about here.The primary determinant of how efficient and scalable your solutions are going to be boils down to how many rows your queries have to touch for any given type of query you submit. Touching in that context includes both the scanning of index tuples and heap access for the rows, the end of the indexed contents of the rows, so be smart with your indexes and only index what you need and design towards the end of needing as few indexes as possible, because those are how you reduce that overhead that your database has to work through to fulfill a particular request or query.Shane: So my advice, at least for an IoT company, is to consider your real-time use cases and your historical use cases and consider them separately. What we found out is we treated everything the same. Everything went through the same pipeline, and it was just queries that were different. When you're using TimescaleDB—and really any architecture—it's extremely useful to consider those use cases separately and architect them separately. They have different requirements. If you try to shoehorn all data, in real time, into any time-series database, whether it's something awesome like TimescaleDB or something else, it's going to cause problems and it's not going to scale very well. You can do a lot more by separating historical and real-time use cases.We’d like to thank Shane, Brock, and all the folks at Density for sharing their experience with TimescaleDB in measuring and optimizing real estate usage.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/density-measures-large-real-estate-portfolios-using-timescaledb/
2022-05-24T14:45:19.000Z,How NLP Cloud Monitors Their Language AI API,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Julien Salinas, full-stack developer, founder, and chief technology officer (CTO) at NLP Cloud, talks about how his team is building an advanced API to perform natural language processing (NLP) tasks while taking care of the complexity of AI infrastructures for developers.About the CompanyNLP Cloudis an advanced API for text understanding and generation in production. The most recent AI models are easy to use on NLP Cloud (GPT-NeoX 20B, GPT-J, Bart Large, among others). Thanks to the API, you can perform all kinds of natural processing language processing tasks: text summarization, paraphrasing, automatic blog post generation, text classification, intent detection, entity extraction, chatbots, question answering, and much more!Several API clients are available, so you can easily add language AI to your application in Python, Go, JavaScript, Ruby, and PHP. You can also train or fine-tune your own AI model on NLP Cloud or deploy your in-house models.Today, more than 20,000 developers and data scientists use NLP Cloud successfully in production! They love NLP Cloud because they don’t want to deal with MLOps (DevOps for machine learning) by themselves. NLP Cloud takes care of the complex infrastructure challenges related to AI (GPU reliability, redundancy, high availability, scaling, etc.).About the TeamMy name is Julien Salinas, and I’m a full-stack developer, founder, and CTO of NLP Cloud. Our company has a team of six high-level engineers skilled in NLP, DevOps,  and low-level optimization for machine learning.The team works hard to provision state-of-the-art NLP models like GPT-NeoX (equivalent to OpenAI’s GPT-3) and make sure that these models run reliably in production and at an affordable cost.About the ProjectA summarization task performed by the NLP Cloud APIWe realized we needed a time-series database when our users asked us for a pay-as-you-go planfor GPT-J, one of our open-source NLP models. They wanted to be charged based on the number of words they’re generating, the same way OpenAI does with their GPT-3 API. Our users also wanted to monitor their usage through their NLP Cloud dashboard.So, we started implementing TimescaleDB to log the following:The number of API calls per user and API endpointThe number of words sent and generated per user by GPT-J and GPT-NeoXThe number of characters sent and received by our multilingual add-onWe had two main requirements:Writing the data had to be very fast in order not to slow down the APIQuerying the data had to be easy for our admins to quickly inspect the data when needed and easily show the data to our customers on their dashboardChoosing (and Using!) TimescaleDB✨Editor’s Note:Not that you really need them, but here are nine reasons to choose TimescaleDB vs. InfluxDB or AWS Timestream.I found out about Timescale by looking for InfluxDB alternatives. I found the Telegraf, Influx, and Grafana (TIG) stack quite complex, so I was looking for something simpler.“TimescaleDB is a cornerstone of our pay-as-you-go plans”The top factors in my decision for Timescale were the following:Easy data downsampling thanks to continuous aggregatesPostgreSQL ecosystem: no need to learn something new, and we were all already skilled in SQL and PostgreSQL, so it saved us a lot of time and energyWe use TimescaleDB behind ournatural language processing APIto track API usage. Based on that, we can do analytics on our API and charge customers depending on their consumption. TimescaleDB is a cornerstone of our pay-as-you-go plans. Most of our users select such plans.If you want to see how we do it, I detailedhow we use TimescaleDB to track our API analyticsin a previous blog post.“The greatest TimescaleDB feature for us is the ability to automatically downsample data thanks to continuous aggregates”Before using TimescaleDB, we did very naive analytics by simply logging every API call into our main PostgreSQL database. Of course, it had tons of drawbacks. We had always known it would be a temporary solution as long as the volume of API calls remained reasonably low (right after launching the API publicly), and we quickly switched to TimescaleDB as soon as possible.We also evaluated a TIG solution (InfluxDB) but found that the complexity was not worth it. If TimescaleDB did not exist, we would maybe stick to a pure log-based solution backed by Elasticsearch.Current Deployment and Future PlansWe use TimescaleDB as a Docker container automatically deployed by our container orchestrator. Two kinds of applications insert data into TimescaleDB: Go and Python microservices. To visualize the data, we’re using Grafana.The greatest TimescaleDB feature for us is the ability to automatically downsample datathanks to continuous aggregates. We’re writing a lot of data within TimescaleDB, so we can’t afford to keep everything forever, but some high-level data should be kept forever. Before that, we had to develop our own auto-cleaning routines on PostgreSQL: it was highly inefficient, and some of our read queries were lagging. It’s not the case anymore.✨Editor’s Note:Learn how you can proactively manage long-term data storage with downsamplingorread our docs on downsampling.The NLP Cloud API is evolving very fast. We are currently working hard on multi-account capabilities: soon, our customers will be able to invite other persons from their team and manage multiple API tokens.In the future, we also plan to integrate severalnew AI modelsand optimize the speed of our Transformer-based models.Advice and ResourcesWe recommend Timescale to any development team looking for a time-series solution that is both robust and easy to deal with. Understandably, most developers don’t want to spend too much time implementing an analytics solution. We found that TimescaleDB was simple to install and manage for API analytics, and it scales very well.TheTimescaleDB docsare a very good resource. We didn’t use anything else.My advice for programmers trying to implement a scalable database strategy? Don’t mix your business database or online transaction processing (OLTP) with your online analytical processing or analytics database (OLAP).It’s quite hard to efficiently use the same database for both day-to-day business (user registration and login, for example) and data analytics. The first one (OLTP) should be very responsive if you don’t want your user-facing application to lag, so you want to avoid heavy tasks related to data analytics (OLAP), as they are likely to put too much strain on your application.Ideally, you want to handle data analytics in a second database that is optimized for writes (like TimescaleDB) and is perfectly decoupled from your OLTP database. The trick then is to find a way to properly move some data from your OLTP database to your OLAP database. You can do this through asynchronous extract, transform, and load (ELT) batch jobs, for example.We’d like to thank Julien and his team at NLP Cloud for sharing their story and writing a blog post on how the NLP Cloud Team uses TimescaleDB to track their API analytics.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-nlp-cloud-monitors-their-language-ai-api/
2021-08-25T14:04:30.000Z,How Messari Uses Data to Open the Cryptoeconomy to Everyone,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Adam Inoue, Software Engineer at Messari, joins us to share how they bring transparency to the cryptoeconomy, combining tons of data about crypto assets with real-time alerting mechanisms to give investors a holistic view of the market and ensure they never miss an important event.Messariis a data analytics and research company on a mission to organize and contextualize information for crypto professionals. Using Messari, analysts and enterprises can analyze, research, and stay on the cutting edge of the crypto world – all while trusting the integrity of the underlying data.This gives professionals the power to make informed decisions and take timely action. We are uniquely positioned to provide an experience that combines automated data collection (such as our quantitativeasset metricsandcharting tools) withqualitative researchandmarket intelligencefrom a global team of analysts.Our users range from some of the most prominent analysts, investors, and individuals in the crypto industry to top platforms like Coinbase, BitGo, Anchorage, 0x, Chainanalysis, Ledger, Compound, MakerDAO, andmany more.About the teamI have over five years of experience as a back-end developer, in roles where I’ve primarily focused on high-throughput financial systems, financial reporting, and relational databases to support those systems.After some COVID-related career disruptions, I started at Messari as a software engineer this past April (2021). I absolutely love it. The team is small, but growing quickly, and everyone is specialized, highly informed, and at the top of their game. (Speaking of growing quickly,we’re hiring!)We’re still small enough to function mostly as one team. We are split into front-end and back-end development. The core of our back-end is a suite of microservices written in Golang and managed by Kubernetes, and I - along with two other engineers - “own” managing the cluster and associated services. (As an aside, another reason I love Messari: we’re a fully remote team: I’m in Hawaii, and those two colleagues are in New York and London. Culturally, we also minimize meetings, which is great because we’re so distributed,andwe end up with lots of time for deep work.)From a site reliability standpoint, my team is responsible for all of the back-end APIs that serve the live site, ourpublic API, our real-time data ingestion, the ingestion and calculation of asset metrics, and more.So far, I’ve mostly specialized in the ingestion of real-time market data – and that’s where TimescaleDB comes in!About the projectMuch of our website is completely free to use, but we haveProandEnterprisetiers that provide enhanced functionality. For example, our Enterprise version includesIntel, a real-time alerting mechanism that notifies users about important events in the crypto space (e.g., forks, hacks, protocol changes, etc.) as they occur.We collect and calculate a huge catalog ofcrypto-asset metrics, like price, volume, all-time cycle highs and lows, and detailed information about each currency. Handling these metrics uses a relatively low proportion of our compute resources, while real-time trade ingestion is a much more resource-intensive operation.Our crypto price data is currently calculated based on several thousand trades per second (ingested from partners, such asKaikoandGemini), as well as our own on-chain integrations withThe Graph. We also keep exhaustive historical data that goes as far back as the dawn of Bitcoin. (You can read more about the history of Bitcoinhere.)Messari dashboard, with data available atmessari.iofor freeOur data pipelines are the core of the quantitative portion of our product – and are therefore mission-critical.For our site to be visibly alive, the most important metric is our real-time volume-weighted average price (VWAP), although we calculate hundreds of other metrics on an hourly or daily basis. We power our real-time view through WebSocket connections to our back-end, and we keep the latest price data in memory to avoid having to make constant repeated database calls.Everything “historical” - i.e., even as recently as five minutes ago - makes a call to our time-series endpoint.Any cache misses there will hit the database, so it’s critical that the database is highly available.We use the price data to power the quantitative views that we display on our live site, and we also directly serve our data to API users. Much of what we display on our live site is regularly retrieved and cached by a backend-for-frontend GraphQL server, but some of it is also retrieved by HTTP calls or WebSocket connections from one or more Go microservices.The asset view of Messari dashboard, showing various price stats for a specific currency.The accuracy of our data is extremely important because it’s public-facing and used to help our users make decisions. And, just like the rest of the crypto space, we are also scaling quickly, both in terms of our business and the amount of data we ingest.Choosing (and using!) TimescaleDBWe’re wrapping up a complete transition to TimescaleDB fromInfluxDB. It would be reasonable to say that we used InfluxDB until it fell over; we asked it to do a huge amount of ingestion and continuous aggregation, not to mention queries around the clock, to support the myriad requests our users can make.Over time, we pushed it enough that it became less stable, so eventually, it became clear that InfluxDB wasn’t going to scale with us. Thus,Kevin Pyc(who served as the entire back-end “team” until earlier this year) became interested in TimescaleDB as a possible alternative.The pure PostgreSQL interface and impressive performance characteristics sold him on TimescaleDB as a good option for us.From there, the entire tech group convened and agreed to try TimescaleDB. We were aware of its performance claims but needed to test it out for ourselves, for our exact use case. I began by reimplementing our real-time trade ingestion database adapter on TimescaleDB – and on every test, TimescaleDB blew my expectations out of the water.The most significant aspects of our system are INSERT and SELECT performance.INSERTs of real-time trade data are constant, 24/7, and rarely dip below 2,000 rows per second. At peak times, they can exceed 4,500—and, of course, we expect this number to continually increase as the industry continues to grow and we see more and more trades.SELECT performance impacts our APIs’ response time for anything we haven’t cached; we briefly cache many of the queries needed for the live site, but less common queries end up hitting the database.When we tested these with TimescaleDB, both of our SELECT and INSERT performance results flatly outperformed InfluxDB. In testing, even thoughTimescale Cloudis currently only located in us-east-1 and most of our infrastructure is in an us-west region, we saw an average of ~40ms improvement in both types of queries. Plus, we could batch-insert 500 rows of data, instead of 100, with no discernible drop in execution time relative to InfluxDB.These impressive performance benchmarks, combined with the fact that we can use Postgres with foreign key relationships to derive new datasets from our existing ones (which we weren’t able to do with InfluxDB) are key differentiators for TimescaleDB.✨Editor’s Note:For more comparisons and benchmarks, seehow TimescaleDB compares to InfluxDB, MongoDB, AWS Timestream, and other time-series database alternativeson various vectors, from performance and ecosystem to query language and beyond. For tips on optimizing your database insert rate, see our13 ways to improve PostgreSQL insert performanceblog post.We are also really excited about continuous aggregates. We store our data at minute-level granularity, so any granularity of data above one minute is powered by continuous queries that feed a rollup table.In InfluxDB-world, we had a few problems with continuous queries: they tended to lag a few minutes behind real-time ingestion, and, in our experience, continuous queries would occasionally fail to pick up a trade ingested out of order—for instance, one that’s half an hour old—and it wouldn’t be correctly accounted for in our rollup queries.Switching these rollups to TimescaleDB continuous aggregates has been great; they’re never out of date, and we can gracefully refresh the proper time range whenever we receive an out-of-order batch of trades or are back-filling data.At the time of writing, I’m still finalizing our continuous aggregate views—we had to refresh them all the way back to 2010! — but all of the other parts of our implementation are complete and have been stable for some time.✨Editor’s Note:Check out thecontinuous aggregates documentationand followthe step-by-step tutorialto learn how to utilize continuous aggregates for analyzing the NFL dataset.Current deployment & future plansAs I mentioned earlier, all of the core services in our back-end are currently written in Go, and we have some projects on the periphery written in Node or Java. We don't currently need to expose TimescaleDB to any project that isn't written in Go. We useGORMfor most database operations, so we connect to TimescaleDB with agorm.DBobject.We try to use GORM conventions as much as possible; for TimescaleDB-specific operations likemanaging compression policiesor thecreate_hypertablestepwhere no GORM method exists, we write out queries literally.For instance, we initialize our tables usingrepo.PrimaryDB.AutoMigrate(repo.Schema.Model), which is a GORM-specific feature, but we create new hypertables as follows:res := repo.PrimaryDB.Table(tableName).Exec(
		fmt.Sprintf(""SELECT create_hypertable('%s', 'time', chunk_time_interval => INTERVAL '%s');"",
			tableName, getChunkSize(repo.Schema.MinimumInterval)))Currently, our architecture that touches TimescaleDB looks like this:The current architecture diagram.We use Prometheus for a subset of our monitoring, but, for our real-time ingestion engine, we’re in an advantageous position: the system’s performance is obvious just by looking at our logs.Whenever our database upsert backlog is longer than a few thousand rows, we log that with a timestamp to easily see how large the backlog is and how quickly we can catch up.Our backlog tends to be shorter and more stable with TimescaleDB than it was previously – and opur developer experience has improved as well.Speaking for myself, I didn’t understand much about our InfluxDB implementation’s inner workings, but talking it through with my teammates, it seems highly customized and hard to explain from scratch. The hosted TimescaleDB implementation with Timescale Cloud is much easier to understand, particularly because we can easily view the live database dashboard, complete with all our table definitions, chunks, policies, and the like.Looking ahead, we have a lot of projects that we’re excited about! One of the big ones is that, with TimescaleDB, we’ll have a much easier time deriving metrics from multiple existing data sets.In the past, because InfluxDB is NoSQL, linking time-series together to generate new, derived, or aggregated metrics was challenging.Now, we can use simple JOINs in one query to easily return all the data we need to derive a new metric.Many other projects have to remain under wraps for now, but we think TimescaleDB will be a crucial part of our infrastructure for years to come, and we’re excited to scale with it.Getting started advice & resourcesTimescaleDB is complex, and it's important to understand the implementation of hypertables quickly. To best benefit from TimescaleDB’s features, you need to think about how to chunk your hypertables, what retention and compression policies to set, and whether/how to set up continuous aggregates. (Particularly with regard to your hypertable chunk size, because it's hard to change that decision later.)In our case, the “answers” to three of these questions were addressed from our previous InfluxDB setup: compress after 48 hours (the maximum time in the past we expect to ingest a trade); retain everything; and rollup all of our price and volume data into our particular set of intervals (5m, 15m, 30m, 1h, 6h, 1d, and 1w).The most difficult part was understanding how long our chunks should be (i.e., setting ourchunk_time_intervalon each hypertable). We settled on one day, mostly by default, with some particularly small metrics chunked after a year instead.I’m not sure these decisions would be as obvious for other use cases.✨Editor’s Note:We’ve put togethera hypertable best practices guideto help you get started (including tips on how to size your chunks andcheck your chunk size).Explore the roadmap on GitHubfor future plans on compression.In summary, the strongest advantages of TimescaleDB are its performance and pure Postgres interface. Both of these make us comfortable recommending it across a wide range of use cases. Still, the decision shouldn’t be cavalier; we tested Timescale for several weeks before committing to the idea and finishing our implementation.We’d like to thank Adam and all of the folks at Messari for sharing their story, as well as for their effort to lower the barriers to investing in crypto assets by offering a massive amount of crypto-assets metrics and a real-time alerting mechanism.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Lucie Šimečková), and we’ll go from there.Additionally, if you’re looking for more ways to get involved and show your expertise, check out theTimescale Heroesprogram.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-messari-uses-data-to-open-the-cryptoeconomy-to-everyone/
2022-11-25T16:00:00.000Z,How to Reduce Query Cost With a Wide Table Layout in TimescaleDB,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Florian Herrengt, co-founder ofNocodelytics,shares how he tracks, records, and monitors millions of user interactions per day to build dazzling analytics dashboards for website building and hosting platformWebflow. And the best part? The team is making queries up to six times less costly by using a wide table layout.About the CompanyA Nocodelytics dashboardNocodelyticsis an analytics product built specifically for the website building and hosting companyWebflow. We are processing millions of events from various websites and turning them into insightful dashboards.We’re a close-knit team of three based in London. We are two co-founders,Sarwechand myself (Florian), and the team recently expanded whenAlexjoined us as a frontend developer. You can find us on theWebflow Marketplace.As our company is growing fast, we found that a quick, reliable database is vital for our company to grow and thrive.About the ProjectOur goal is to build the number one analytics platform for Webflow.Like other analytics tools, we provide users with a tracking script that they add to their Webflow site. However, because of the nature of Webflow’s audience, we have to do things quite differently from other analytics tools—which presents challenges that no other analytics tool faces.First, one of the things that we do that adds complexity is that we automatically track every event a user does. Whether it’s clicking on a button or link, interacting with a form, or viewing a page, we need to be able to track all of this information with minimal impact on accuracy.Adding a new metric to the Nocodelytics dashboardWe also track events tied to the Webflow Content Management System (CMS) and other third-party tools likeJetboost,Wized,Memberstack, andOutseta, which we automatically integrate with and track.So, we tap into the power of the CMS and the Webflow ecosystem to record how users interact. We then output these interactions into valuable insights for our analytics customers. This means we need to be able to record and ingest millions of events into our database per day without it crashing down. Some of our customers will get publicity and see huge spikes in traffic, so we need to be able to handle this too.Second, we provide our customers with a simple-to-use and customizable dashboard. This allows them to create metrics that go deep and answer almost any question (What are the most popular jobs on my site? How many signups came from Google? Which contact button is most effective?).To do this, we’re building a metric creator that is simple to use on the frontend but complex on the backend, with some advanced SQL being done to return the right answers depending on the question asked. So it’s important that we have the right tool to help us with this.Third, when our customers view their dashboard and look at the metrics, even a few seconds’ wait can cause frustration. As our customers can have several metrics on the dashboard at any time—some with fairly complex queries—there’s a lot of pressure on our database to read the data, crunch the numbers, and return the result quickly.On top of that, we also allow our customers to share and embed their dashboard onto a site, which means the number of users viewing the metrics goes up, and the number of read requests can increase at any time.Choosing (and Using!) TimescaleDBFirst, let’s talk about the previous setup we had and what problems this resulted in.Like many other companies,Nocodelyticsstarted with PostgreSQL. In the beginning, it worked. But the size of the database grew very, very fast. Eventually, with millions of rows, our dashboards became sluggish. Queries for customers with a lot of traffic would take several minutes or even time-out.As we needed a solution as quickly as possible, I had three things in mind when looking for an alternative to PostgreSQL:It had to be quick to learn.The change needed to involve a minimal amount of code.The migration path had to be simple.My first choice wasClickHouse, which seems to have better performance than Timescale for our use case—but keep reading as there's more to it.Not everything was great about ClickHouse: It does a lot, which can get confusing, and I’d rather stick with PostgreSQL, which I’ve used for years and know works.Amazon Athenawas another good option. It's serverless and queries compressed data directly from S3 (which Timescale is now offering in private beta too). It did have some weird limitations (e.g., a maximum of 20 concurrent queries, no way to update data, and dynamic partition columns must always be included in queries), which I found out the hard way. At that point, I was worried about the next issue I’d find, and I lost confidence in the product.Finally,InfluxDB. I spent a few hours with it, but it’s too different from what we already had. The migration would take forever.Also, I must stress that I had never heard about those tools before. I either worked on large projects with big money, where we used Redshift/BigQuery or specialized, small-scale projects, where the smallest PostgreSQL instance would be enough.I was about to use ClickHouse before I came across Timescale by chance while browsing databases.It’s just PostgreSQLThere you have it. The best feature of TimescaleDB: it's all PostgreSQL, always has been. All your tools, all the existing libraries, and your code already work with it. I’m using TimescaleDB because it’s the same as PostgreSQL but magically faster.Whatever technology is behind TimescaleDB, it’s truly impressive. Since theWebflow Conf, we have been inserting more than a million rows per day (without optimizations) in our tiny 8 GB memory instance. Sometimes, we have 3 K/IOPS. PostgreSQL would struggle. It’s like pulling an elastic until it snaps—but it never does, and we barely scratched the surface of what it can do. Also, the community is really nice.“I’m using TimescaleDB because it’s the same as PostgreSQL but magically faster""So, in sum, Timescale was a drop-in replacement that solved most of our issues. I installed the extension,created a hypertable, and everything became magically fast.✨Editor’s Note: Want to get started with TimescaleDB?Check out our documentation.But as I was reading the Timescale documentation, I realized it could be faster. A lot faster.Relational vs. Wide Table LayoutWhen you first learn about relational databases, you learn how to normalize your data with multiple tables and foreign key references. That’s a good, flexible way to store your data. However, it can be an issue when dealing with a large amount of data.That’s where the wide table layout becomes useful.Normalized data vs. Wide tableThe idea is to trade storage and schema flexibility for query performance by reducing the number ofJOINs. But this doesn’t stop you from combining both. You can still add foreign keys to a wide table.You will end up using more storage, but you can mitigate it with TimescaleDB’s compression.✨Editor’s Note:Learn how to save space using compression.Show time: Setting Up the SchemaLet’s create the above schema with relationships and insert dummy data:-- Sequence and defined type
CREATE SEQUENCE IF NOT EXISTS events_id_seq;
CREATE SEQUENCE IF NOT EXISTS countries_id_seq;
CREATE SEQUENCE IF NOT EXISTS browsers_id_seq;
CREATE SEQUENCE IF NOT EXISTS devices_id_seq;
 
-- Table Definition
CREATE TABLE ""public"".""countries"" (
   ""id"" int4 NOT NULL DEFAULT nextval('countries_id_seq'::regclass),
   ""name"" varchar,
   PRIMARY KEY (""id"")
);
 
CREATE TABLE ""public"".""browsers"" (
   ""id"" int4 NOT NULL DEFAULT nextval('browsers_id_seq'::regclass),
   ""name"" varchar,
   PRIMARY KEY (""id"")
);
 
CREATE TABLE ""public"".""devices"" (
   ""id"" int4 NOT NULL DEFAULT nextval('devices_id_seq'::regclass),
   ""name"" varchar,
   PRIMARY KEY (""id"")
);
 
CREATE TABLE ""public"".""events"" (
   ""id"" int4 NOT NULL DEFAULT nextval('events_id_seq'::regclass),
   ""name"" varchar,
   ""value"" int,
   ""country_id"" int,
   ""browser_id"" int,
 
   ""device_id"" int,
   PRIMARY KEY (""id"")
);
create index events_country_id on events(country_id);
create index events_browser_id on events(browser_id);
create index events_device_id on events(device_id);
create index countries_name on countries(name);
create index browsers_name on browsers(name);
create index devices_name on devices(name);Then create our new wide table:create table events_wide as
   select
       events.id as id,
       events.name as name,
       events.value as value,
       countries.name as country,
       browsers.name as browser,
       devices.name as device
   from events
   join countries on events.country_id = countries.id
   join browsers on events.browser_id = browsers.id
   join devices on events.device_id = devices.id
 
create index events_wide_country on events_wide(country);
create index events_wide_browser on events_wide(browser);
create index events_wide_device on events_wide(device);ResultsNeat. But was it worth it? Well, yes. It would be a lot less interesting to read otherwise. Now that we have our wide table, let’s have a look at the query cost.-- cost=12406.82
explain select devices.name, count(devices.name)
from events
join countries on events.country_id = countries.id
join browsers on events.browser_id = browsers.id
join devices on events.device_id = devices.id
where browsers.name = 'Firefox' and countries.name = 'United Kingdom'
group by devices.name order by count desc;
 
-- cost=2030.21
explain select device, count(device)
from events_wide
where browser = 'Firefox' and country = 'United Kingdom'
group by device order by count desc;This is a significant improvement. The same query is six times less costly. For a dashboard with dozens of metrics, it makes amassivedifference.You can find the full SQLhere.Future PlansTimescale is packed with amazing features we want to start using. Things liketime_bucket_gapfill()orhistogram().I didn't dive into it yet, but theTimescale Toolkitseems to have a lot of valuable functionality, such asapproximate count distinctsorfunction pipelines, which we can’t wait to try out.We also want to see howcontinuous aggregatescould help us relieve some pressure on the database.Our goal is to keep growing and scaling the number of events we store. We will soon leveragetablespacesanddata tiering to save on storage space. We’re keen to further optimize and use TimescaleDB to help as we grow towards handling billions of rows!June 2023 update:We’re now dealing with more than 500 GB of data, and those wide tables just aren’t efficient anymore.So, we’ve gone ahead and separated the table again. We’re executing a count query first, retrieving the ids, then running another query for the labels. Essentially, it’s a two-query process.TimescaleDB is row-based and our wide table is heavy on strings. As a result, we’re hitting I/O limits. This wasn’t a problem before because we’re using a fast SSD and had fewer rows per site, but now with the data volume, it’s a different story.In retrospect, choosing the wide table structure at that time was good. It accelerated our development pace significantly. We centralized all the events, simplifying our queries for quite some time. Plus, it enabled us to compress all our data without effort. Looking back, it was a beneficial strategy for that stage of our project.We’d like to thank Florian and the folks at Nocodelytics for sharing their story on tracking millions of user events while reducing their query cost using TimescaleDB. Stay tuned for an upcoming dev-to-dev conversation between Florian and Timescale’s developer advocate,Chris Englebert, where they will expand on these topics.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-use-wide-table-model-to-reduce-query-cost-in-timescaledb/
2022-09-20T13:00:00.000Z,Replacing kdb+ With PostgreSQL for Time-Series Forecasting,"I am currently the chief data scientist atRasgo. Before joining Rasgo, I worked as a data scientist and data science leader for over 10 years across various fields. I received my Ph.D. in Systems and Industrial Engineering with a focus on decision-making under uncertainty and probability modeling from the University of Arizona. After my Ph.D., I was an assistant professor of Applied Mathematics at Towson University, where I began working on data science problems.As my career progressed, I found myself doing less and less technical work in data science and more time spent on team building, interacting with management and customers, and basically talking about data science instead of doing data science. This culminated in 2016 when I took a role withDataRobotto help their prospects and customers use DataRobot’s machine learning platform to implement their data science initiatives successfully.While this was, and still is, fulfilling work, I missed the technical work that originally drew me into data science. DataRobot encouraged us to do this (and leverage their product). In short, I became the DataRobot Marketing team's go-to content writer about sports analytics and betting. As part of this, I would build machine learning models to predict the outcome of sporting events. I regularly wrote blogs on my predictions for Grand Slam tennis, the Super Bowl (and NFL playoffs), the Champions League knockout stage, and the Premier League.Because of this and the contacts developed over the years as a data scientist, multiple people have asked me to advise and even build proofs of concept of machine learning pipelines to automate both sports betting predictions and market models (from financial markets to sports betting markets in aid of monetizing these sports predictions).Andrew’setu package on GitLabIn my spare time, I have been building proofs of concept of machine learning pipelines to pull relevant data (this could be sports statistics, historical sports betting odds, or financial market pricing data), use machine learning to make predictions, and share those predictions back to the organization to help them make decisions or bets.Using PostgreSQL, I would write the original data, the engineered features for modeling, andthe final predictions into the database for model monitoring and future model development.In all of these cases, the underlying data is event data and has time-series characteristics. This means that as a data scientist, I was interested in capturing how the data changed over time. For example, when modeling a sports betting market, I was interested in how the betting line or odds would change over time. Was it rising, falling, oscillating, etc.? Capturing this information as machine learning features was key to my models performing well. I typically used a Python library calledtsfreshto build these features.✨Editor's Note:Do you know what time-series data is?The answer is in this blog post.As the amount of data grew, I spent most of the time in the pipeline, pulling data from the database, building the features, and pushing the results back to the database.Understanding kdb+As I was first working on these sorts of problems in my spare time, DataRobot asked me to lead their go-to-market efforts in the sports and entertainment space. In this new role, I began talking with several Formula 1 teams about possibly using DataRobot in their workflows.“During the investigation of TimescaleDB, I realized I could use PostgreSQL’s support of custom extensions built in Python to allow me to call tsfresh functions directly within the database without the expense of moving the data out or pulling it back in”As part of these conversations, I learned aboutKdb+ from KX Systems. This is a high-performance columnar time-series database used by F1 teams to store and process sensor data collected from their cars (and simulations). Financial institutions also use it to store and process market data. More intriguing from my point of view was that Kdb+ included a reimplementation of tsfresh within the database.I understood the power of Kdb+, but its price was well out of my budget for these proofs of concept. Still, I liked the idea of a database that supported time-series data and could process it efficiently.Using Timescale'stime_bucket()and custom extensions for easier time-series aggregationsI have been using PostgreSQL for years and including it in the technology stack I used in these proofs of concept. While other databases supported time series, TimescaleDB combined the familiarity of PostgreSQL, making it the clear choice.During the investigation of TimescaleDB, I realized I could use PostgreSQL’s support of custom extensions built in Python to allow me to call tsfresh functions directly within the database without the expense of moving the data out or pulling it back in.In addition,TimescaleDB’s time_bucket() functionwould allow me to perform my time-series aggregations for arbitrary time intervals (eg., on a daily or weekly basis) as opposed to estimating the time period from a set number of rows. This was a huge improvement over using either tsfresh in Python (where it worked at a set number of rows) or a window function in PostgreSQL.I then implemented all of the tsfresh functions as custom extensions in Python and built custom types to pass the necessary information into the function and custom aggregations to perform the aggregations to get the data ready for the time-series feature.As expected, there were significant benefits to moving this processing into the database. In addition to eliminating the time to extract and insert the data, the database handled running these calculations in parallel much more simply than trying to manage them within Python.“By releasing this as open source, I hope to help anyone [...] working on any sort of time-series problem to be able to fully leverage the power of TimescaleDB to manage their time-series data”It was not without its drawbacks, however. First, PL/Python is untrusted in PostgreSQL and limits the types of users that could use these functions. More importantly, each time the function was called, the Python interpreter was started, and the tsfresh library and its dependencies were loaded. This meant that most of the processing time was spent on getting ready to perform the calculations instead of the actual calculations. The speedup was significantly less than expected.At this point, I saw promise in this approach but needed to make it faster. This led me to reimplement the tsfresh functions in C and create custom C extensions for PostgreSQL and TimescaleDB. With these C extensions, I was able to generate a performance improvement between 10-100 times faster than the corresponding performance of the PL/Python version.Reimplementing tsfresh as Custom Functions in PostgreSQLWhen I saw this performance, I felt I needed to clean up my code and make it available to the community as open source. This led to two packages:etu, which contains the reimplementation of tsfresh feature calculators in C, andpgetu, which includes the C extension that wraps etu and makes it available as functions in PostgreSQL and TimescaleDB.As an example, these functions can be called in a SELECT statement in TimescaleDB as:SELECT 
  time_bucket(interval '1 months', date) AS month,
  absolute_sum_of_changes(amount, date) AS amount_asoc,
  approximate_entropy(amount, date, 1, 0.1) AS amount_ae,
  fft_aggregated(amount, date, ARRAY['centroid', 'variance', 'skew', 'kurtosis']) as ammount_fft_agg,
  median(amount) as median_amount,
  mode(amount) as mode_amount
FROM transactions
GROUP BY month
ORDER BY month;Here’s another example: this is looking at the power output of a windmill based on wind speed.SELECT TIME_BUCKET(interval '1 hour, time) AS hour,
autocorrelation(windspeed, time, 1) AS windspeed_autocorrelation_1,
autocorrelation(output, time, 1) AS output_autocorrelation_1,
count_above_mean(output) AS output_cam,
count_below_mean(output) AS output_cbm,
energy(windspeed, time) AS windspeed_energy,
fft_coefficient(windspeed, time, ARRAY[1, 1, 2, 2], ARRAY['real', 'imag', 'real', 'imag'] AS windspeed_fft_coeff_json,
number_cwt_peaks(windspeed, time, 5) AS windspeed_number_wavelet_peaks_5,
number_cwt_peaks(output, time, 5) AS output_number_wavelet_peaks_5
FROM sensor_data
GROUP BY hour
ORDER BY hour;This makes it really easy to perform time-series feature engineering directly in the database.By releasing this as open source, I hope to help anyone working as a data scientist on time-series-like problems quickly generate features directly within their database and anyone else working on any sort of time-series problem to be able to fully leverage the power of TimescaleDB to manage their time-series data.As an open-source project, I hope to see people benefit from this in their personal projects and hopefully find enough value to be interested in helping improve both of these libraries going forward.Adding Additional Custom Functions to PostgresThis project is the culmination of about one year of part-time development work. I spent the first third building a version using Python, the following third understanding how to build C extensions, writing proof of concept versions of some of the functions, and testing them to determine if the speedup was worthwhile (it was a 10x to 100x in extreme cases).Regarding feedback, I have just released it and received a small amount of positive press on thePostgreSQL subredditbut limited feedback from others. I have also shared it with a number of the data scientists in my network, and the feedback has been overwhelmingly positive.Currently,etuandpgetusupport most of the feature calculators in tsfresh. My next step is to implement the approximately 15 calculations that I have not yet finished. Once that is done, I would like to increase the functions these libraries support.If you need more information about this project or want to discuss it, I can be reached and followed onLinkedInandTwitter.We’d like to thank Andrew for sharing his story on how he is creating proofs of concept of machine learning pipelines for time-series forecasting using TimescaleDB.We’re always keen to feature new community projects and stories on our blog. If you have a project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-a-data-scientist-is-building-a-time-series-forecasting-pipeline-using-timescaledb-and-helping-others-perform-time-series-engineering-directly-in-the-database/
2022-02-10T18:03:51.000Z,How Trading Strategy Built a Data Stack for Crypto Quant Trading,"This is an installment of our “Community Member Spotlight” series, where we invite TimescaleDB community members to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Mikko Ohtamaa, CEO at Trading Strategy, joins us to share how they give traders and investors direct access to high-quality trading strategies and real-time control over their assets by integrating market data about thousands of crypto assets, algorithms, profitability simulation, and more into one solution. Thanks to TimescaleDB, they can focus on solving business problems without a need to build the infrastructure layer.Trading Strategyis a protocol for algorithmic trading of crypto-assets in decentralized markets. Cryptocurrency traders and strategy developers can utilize the protocol to easily access and trade on next-generation markets with sophisticated tools that have been traditionally available only for hedge funds.Users don’t need to have a deep understanding of blockchain technology, as the Trading Strategy protocol is designed to make a complex topic easier to understand and approach. This is accomplished by integrating market data feeds for thousands of crypto assets, algorithm development, profitability simulation, trade execution, and smart contract-based treasury management tools into one vertically integrated package.About the teamMy name isMikko Ohtamaa. I am the CEO of Trading Strategy. I have been a software developer for 25 years. For the last decade, I have been CTO of various cryptocurrency and fintech companies. I am also one of the firstEthereum Dappdevelopers and Solidity smart contract auditors.Trading Strategy is a remote-first company with five people. We have offices in London and Gibraltar. We use a combination of remote working tools (Discord, Github, Google Workspace) and more intense get-together sprint weeks to manage the software development.As our work is highly technical, all team members have backgrounds in software development, quantitative finance, or blockchain technologies.About the projectTrading Strategy operates on market data feeds, for which raw data is collected directly from the blockchains. Most market data is heavily time-series, though there are also elements of product catalog features like different trading pairs, tokens, and exchanges.Because of the powerful combination of PostgreSQL and TimescaleDB, we can store the data in a single database,making it simple for software developers to build on top of this and this, in turn, saves us a lot of software development costs.We have two kinds of workloads:historical datasets, that are challenging sizewise, andreal-time data feedsfor algorithms that are challenging latency-wise. TimescaleDB offers vertical scaling as a data lake,  but also offers continuous real-time aggregations for time-series data, making it a good fit for real-time needs.✨Editor’s Note:We’ve put together resources about TimescaleDB’scontinuous aggregatesto help you get started.Data from TimescaleDB is feeding bothon our market information websiteand feeds, but also on the trading algorithms themselves, which make the trading decisions based on the data input. Our applications include OHLCV, or so-called candle charts, market summaries information like daily top trades and liquidity, and risk information for technical trade analysis.Collage of charts from Trading Strategy websiteChoosing (and using!) TimescaleDBPostgreSQL has been the open-source workhorse of databases for the last three decades and offers the most well-known, solid, foundation to build your business on.We chose TimescaleDB over other time-series databases because of its solid PostgreSQL foundation, easy, out-of-the-box functionality, and true open-source nature with an active TimescaleDB community.Moreover, TimescaleDB comes with well-documented code examples on how to use it for stock-market chart data, allowing us to take these examples and build our first MVP based on TimescaleDB example code.✨Editor’s Note:Check out ourthree-step tutorialto learn how to collect, store, and analyze intraday stock data.For example, we heavily utilize the continuous aggregate view feature of TimescaleDB, to upsample our 1-minute candle data to 15 minutes, 1 hour, and daily candles.We can fully offload this work to TimescaleDB, with only a minimal 10-20 lines of SQL code describing how to upsample different columns.Current deployment & future plansBesides TimescaleDB, our other major components in the software development stack are Svelte/SvelteKitweb frontend framework and Python, Pyramid, andSQLAlchemybackend.I invite everyone evaluating TimescaleDB toread our blog post about our software architecture.The current architecture diagramAt the moment, we have trading data from 1000 decentralised exchanges (aka dexes), from three blockchains (Ethereum, Polygon and Binance Smart Chain), featuring 800k trading pairs. For reference,NASDAQhas only 3000 trading pairs, giving a reference point for the massive diversity of blockchain and cryptocurrency markets! Currently, we are fitting everything on one 1 TB database, but we are still early on what kind of data points we collect. We expect the database to grow dozens of terabytes over the next year.Architecture diagram of trading data sourcesTrading Strategy is completing its seed round. So far, the team has been lean. We expect to start growing as a business now, as our business is finding a product-market fit. We are looking to launch user-accessible trading strategies later this year, as soon as we are confident the software stack is well-behaving and related smart contracts are secure.We are at the bleeding edge of blockchain technology. Many of the components we built and many of the problems we solve we do as the first in the world.TimescaleDB allows us to focus on solving these business problems without us needing to build the infrastructure layer ourselves.🗯️If you are interested in crypto trading strategies, please come to ask any questions in ourpublic Discord chat.Advice & resourcesIf you are generally interested in algorithmic trading and machine-based solutions on financial markets, pleaseread our announcement blog postto learn about our vision for decentralised finance and decentralised protocol.To see TimescaleDB in action, youcan explore our public real-time API endpoints, view ourreal-time market data charts, ordownload historical market datasets generatedout from TimescaleDB.Trading Strategy contributes heavily to open source. You canstudy our Trading Strategy Python clientand our100% open source SvelteKit frontend.If you have software development questions or questions about trading strategies, please come to ask any questions inour public Discord chat.We also love coffee brewing tips shared by the TimescaleDB. Due to increased brewing activity, our team is now 150% caffeinated.It matters, and is one of the major reasons I came to Timescale to work in#DevRel.And for the record, I'm totally up for answering any coffee questions you want to throw our way. ☕️😉https://t.co/f74kT9eHWF— Ryan Booz (@ryanbooz)January 19, 2022We’d like to thank Mikko and the entire Trading Strategy team for sharing their story! We applaud your effort to give access to high-quality trading strategies to users around the world.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Lucie Šimečková), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-trading-strategy-built-a-data-stack-for-crypto-quant-trading/
2023-03-30T13:00:44.000Z,How Flowkey Scaled Its AWS Redshift Data Warehouse With Timescale,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Nikola Chochkov, lead data scientist atflowkey, shares how his team migrated from Amazon Redshift to TimescaleDB and is driving rapid growth and experimentation by analyzing the users’ behavioral data using TimescaleDB along with Metabase or Jupyter/Rmarkedown Notebooks.About the CompanyFlowkeyis a leading app for learning to play the piano, with over 10 million registered users in more than 100 countries. The company was launched in 2015 and quickly became one of the global leaders in its category.Here is a video from our founder, Jonas Gößling, explaining how it all started.About the TeamWe are a team of around 40 people, with more than 10 of us working in Data and Engineering.We have a Marketing team (responsible for user acquisition, customer relationship management, collaborations, etc.), a Creative team (building all of our visual content, our design, and advertising), Course and Song teams (creating our in-app learning content—e.g., the courses series and the piano renditions of the songs in our library). We also have Customer Support, Product, Engineering, Data, and Operations teams.Many of us take on more than one role, and for many of us, flowkey has been the first significant career step. Not for me personally, though, but still.🙂About the ProjectThe flowkey appWe are a business that depends heavily on analytics for business decision-making. Our experimentation—a major driver of our growth—is powered by the data analysis of user behavioral data (app usage). This data consists of user events, which we track from our product.“We realized we needed to scale our previous Amazon Redshift data warehouse model into a more suitable solution for categorical, time-series data analysis. We found out about TimescaleDB from being part of the PostgreSQL community”For example, a Learn Session starts at a given timestamp for a user, and we record this event.✨Editor’s Note:Time-series data is a sequence of data points collected over time intervals, allowing us to track changes over time. To learn more, readWhat Is Time-Series Data (With Examples).When we launch a new feature, we typically A/B test it and evaluate its impact based on measuring key performance indicators (KPIs), which are predefined for the test. Every day, we receive millions of incoming events, tracked around our product, in the format:(user_id, event, timestamp, properties)Thepropertiesfield is a schemaless object that depends on the particular event type that we track from the app. For example, a Learn Session event would have asongIdand alearnMode, while a Subscription Offer interaction event would have aproductId, etc.Choosing (and Using!) TimescaleDBWith time, we realized we needed to scale our previous Amazon Redshift data warehouse model into a more suitable solution for categorical, time-series data analysis.We found out about TimescaleDB from being part of the PostgreSQL community, and when we were faced with the problem at hand, it was a natural way forward for us.After doing our research, we realized that TimescaleDB suited our needs perfectly. Here's a list of our arguments:Our data analysts are well-versed in SQL and PostgreSQL.The events’ raw data is available in a PostgreSQL schema alongside all our other business intelligence data.TimescaleDB is an actively developed, open-source solution. It allowed us to deploy it on our self-hosted PostgreSQL data warehouse.A TimescaleDB hypertable model would allow us to accommodate the schemaless JSON structure of our events.select event, platform, time, jsonb_pretty(data) from events_today limit 5;
            event             | platform |          time           |                      jsonb_pretty
------------------------------+----------+-------------------------+---------------------------------------------------------
 SONG_OPEN_UI_ELEMENT_CLICKED | ios      | 2022-11-03 00:00:00.034 | {                                                      +
                              |          |                         |     ""songId"": ""E2kCpqHCwB2xYf7LL"",                     +
                              |          |                         |     ""context"": ""song-cover"",                           +
                              |          |                         |     ""listIndex"": 6,                                    +
                              |          |                         |     ""routeName"": ""Songs"",                              +
                              |          |                         |     ""currentTab"": ""SongsTab""                          +
                              |          |                         | }
 ONBOARDING_QUESTION_VIEWED   | web      | 2022-11-03 00:00:00.145 | {                                                      +
                              |          |                         |     ""context"": ""preferredCategories""                   +
                              |          |                         | }
 SONG_PLAYER_SCROLLED         | ios      | 2022-11-03 00:00:00.157 | {                                                      +
                              |          |                         |     ""level"": 1,                                        +
                              |          |                         |     ""songId"": ""Lui27TDJ4vZBevxzc"",                     +
                              |          |                         |     ""direction"": ""backwards"",                          +
                              |          |                         |     ""songGroupId"": ""vhisAJBkPwvn6tdoq"",                +
                              |          |                         |     ""loopBoundsMs"": null,                              +
                              |          |                         |     ""finalPositionMs"": 0,                              +
                              |          |                         |     ""initialPositionMs"": 24751                         +
                              |          |                         | }
 AB_TEST_VARIANT_ASSIGNED     | ios      | 2022-11-03 00:00:00.249 | {                                                      +
                              |          |                         |     ""variant"": ""CONTROL"",                              +
                              |          |                         |     ""experimentName"": ""player_onboarding_video_09_2022""+
                              |          |                         | }
 ONBOARDING_ANSWER_SUBMITTED  | web      | 2022-11-03 00:00:00.314 | {                                                      +
                              |          |                         |     ""answers"": [                                       +
                              |          |                         |         ""dont-know""                                    +
                              |          |                         |     ],                                                 +
                              |          |                         |     ""context"": ""learningGoals""                         +
                              |          |                         | }TimescaleDB offers a great set of SQL analytical functions.TimescaleDB offers continuous aggregates, which integrate very well with how we do analytics and real-time data monitoring.Data migration and update (e.g., renaming of events or JSON properties) are available.✨Editor’s Note:Faster queries, reduced storage costs, and greater flexibility. Learn more abouthierarchical continuous aggregates.“We use compression, which has cut our disk space usage by 28 percent”Current Deployment & Future PlansOur data warehouse is deployed on self-hosted machines and works well for us. We employ other PostgreSQL extensions that aren't currently supported by the Timescale cloud offering, which was important to us when we launched. These includeMongo FDWandAdjust’siStore extensionfor cohort analysis data storage.✨Editor's Note:We're working on expanding the catalog of PostgreSQL extensions offered in Timescale's cloud offering. Stay tuned!We employ TimescaleDB's awesome data retention (automated through a user action), and thanks to that, our most recent (and more relevant to our analytics) data is available to us on SSD chunks, while historical data is kept on HDDs.Furthermore, we use compression, which has cut our disk space usage by 28 percent. Our data contains JSONB fields, which are difficult to be compressed. We are pretty happy with it, though, so it's a win. 🙂When we do business analytics, we employMetabaseorJupyter/Rmarkdown Notebooksto derive insights. We established a workflow of writing custom continuous aggregates for the duration of experiments, which are then easy to keep and fully deploy or discard, depending on the decision made for the experiment.✨Editor's Note:Learn how toconnect to Timescale from a Jupyter notebookfor better data querying, cleaning, and analysis.This allows us to iterate our experiments quickly and increase the bandwidth of change, which we can successfully bring to the product.RoadmapWe just finished migrating our setup to a more powerful cluster of machines, which allowed us to benefit from the data tiering options mentioned above. Right now, our system is scalable, and we don't expect any major upgrades to this system to come up soon.Advice & ResourcesWe recommend theTimescale documentationas well as theSlack Community.Want more insider tips?Sign up for our newsletterfor more Developer Q&As, technical articles, and tutorials to help you do more with your data. We deliver it straight to your inbox twice a month.We’d like to thank Nikola and all of the folks at flowkey for sharing their story on how they’re improving their online piano lessons by analyzing millions of user events daily using TimescaleDB.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-flowkey-solved-its-time-series-data-problem-by-migrating-from-aws-redshift-to-timescale/
2023-01-30T14:00:44.000Z,How Octave Achieves a High Compression Ratio and Speedy Queries on Historical Data While Revolutionizing the Battery Market,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Nicolas Quintin, head of Data atOctave, shares how the company migrated from AWS Timestream to Timescale in search of a more mature and scalable database and is revolutionizing the battery market by improving the battery systems’ safety and predicting maintenance needs.To do so, the Octave team collects and analyzes millions of data points daily while dramatically saving disk space (their compression ratio is 26.06!) and delivering speedy queries on historical data on client-facing applications. How do they do it? With a little help from Timescale's compression capabilities and continuous aggregates.About the CompanyOctaveis a cleantech company based in Belgium that gives electric vehicle (EVs) batteries a second life. We develop energy storage systems by repurposing usedlithium-ion (Li-ion) batteriesand transforming them into smart, sustainable assets to store the excess wind and solar energy.Batteries from an electric vehicle are typically retired when their usable capacity has decreased to roughly 80 percent. Octave gives these batteries a new life through smart solutions for stationary energy storage, for which demand is rapidly growing. This way, we can save resources and raw materials that would be traditionally used to produce new batteries.The company repurposes batteries from electric vehicles to create these battery cabinetsOctave’s batteries are suitable technologies for small and medium-sized enterprises or industrial sites looking to optimize their energy management or decrease their electricity bill amid the record-high energy prices in Europe.More specifically, Octave’s sustainable energy storage system allows customers to increase their self-consumption and cope with the intermittency of renewable energy sources. It also enables customers to participate actively in the energy markets, becoming more independent from energy suppliers, fossil fuels, and grid operators.About the TeamWe’re currently a team of around 10 people working on everything related to electrical and mechanical design engineering, embedded and software engineering, and business development. We’re growing fast!As the head of data, my role at Octave is to collect data from the edge devices, store them in our databases, develop data pipelines, improve and optimize the battery state algorithms through big data analytics, and present actionable insights in dashboards. Finally, I also ensure that this entire process happens seamlessly.About the ProjectOctave distinguishes itself from traditional battery suppliers by leveraging the plethora of data and measurements from the battery system and building up an extensive history of each battery cell.We handle large streams of battery measurements with clear time-series characteristics: each data point is composed of a timestamp and a value. These data are collected from the battery systems (our IoT edge devices) and sent back to our cloud for further analysis.Among the information we collect, the most basic yet crucial data points are undoubtedly the voltage and temperature measurements from each battery cell in operation.One of Octave’s dashboards made with Grafana and Timescale""We initially used AWS Timestream in the early days of Octave, which at first seemed a natural choice to handle our time-series data since our cloud infrastructure was entirely built in AWS. However, we quickly realized we would need a more widely used, mature, and scalable database solution""The data are streamed back to ourBattery Cloud(our in-house developed cloud platform, hosted on AWS and relying on Timescale databases), where they are crunched and further processed. This allows us the following:Analyze the battery cells’ behavior and degradation based on their history and how they are cycled (the temperature, current levels, and depth of discharge significantly impact the lifetime of batteries!).Improve the safety of the system by immediately detecting anomalies.Implement a data-driven predictive maintenance process. The ultimate goal is to predict when to replace a used battery module, and by doing so, we can extend the lifetime of the entire system. This is a true game-changer for second-life battery systems.✨Editor's Note:Learn what time-series forecasting is, its applications, and its main techniques in this blog post.A diagram of Octave’s Battery CloudChoosing (and Using!) TimescaleDBI have been using PostgreSQL for some time, and we were keen to use tools and stacks we were familiar with for the sake of efficiency. So inevitably, I was immediately interested when I heard about an interesting PostgreSQL extension called TimescaleDB, specifically built to handle time-series data, which we knew we needed since we were dealing with a typical IoT use case.""Timescale has proven to be a key enabler of Octave’s data-driven Battery Cloud technology""We initially used AWS Timestream in the early days of Octave, which at first seemed a natural choice to handle our time-series data since our cloud infrastructure was entirely built in AWS. However, we quickly realized we would need a more widely used, mature, and scalable database solution if we ever wanted to scale our operations and deploy several dozen or hundred second-life battery systems in the field. So, we went looking for alternatives.After some research, Timescale quickly became our preferred option, given itsimpressive compression ratios, lightning-fast queries,unmissable continuous aggregates, friendlycommunityandextensive documentation, and most importantly, its plain PostgreSQL syntax.The cherry on the cake is Timescale's ease of use and user-friendly interface.We went immediately for Timescale because we were looking for a managed service, and Timescale seemed the recommended option and was nicely compatible with our preferred AWS region.✨Editor’s Note:Timescale’s user interface now offers an even friendlier and user-centered experience.The Timescale Team shared their redesign journey and lessons in this blog post.We have found TimescaleDB’s compression ratio to be absolutely phenomenal! We’re currently at a compression ratio of over26, drastically reducing the disk space required to store all our data.A narrow table modelOctave’s compression ratio with TimescaleThe power of continuous aggregates is hard to overstate:they are basically materialized views that are continuously and incrementally refreshedand allow for lightning-fast queries on large historical datasets.I have found thedocumentationto be very clear and abundant. Every feature is very well explained. So it’s very easy toget started with TimescaleDB. And if we have more precise questions specifically related to our use case, we can always rely on our customer success manager or the very reactive Support Team. 🙂✨Editor’s Note:Read how our Support Team raises the bar on hosted database support.Current Deployment & Future PlansMost of our backend software is currently written in Python. We leverage AWS IoT to manage our battery fleet and circulate the data between our edge devices and our battery cloud viaMQTT.We rely on some ETL (extract, transform, load) pipelines interacting with Timescale that extract battery measurements and insert the processed data back into the database. We also use some dashboarding tools, such asGrafanaandStreamlit, as well as API endpoints connected to our Timescale database.It’s no secret that time-series data can grow very fast. We currently store close to one million data points per battery system daily. As we have already sold our first 28 battery cabinets, the size of our database is expected to increase quickly. This is only the start, as we expect to triple or quadruple sales by the end of 2023.So far, we have found Timescale to be powerful, scalable, and pretty well-suited for our application. Timescale has proven to be a key enabler of Octave’s data-driven Battery Cloud technology.We have started to leverage the power of continuous aggregates. For example, continuous aggregates are queried behind the scenes of our customer-facing applications. They enable our clients to quickly and seamlessly inspect and download historical data of their second-life battery systems.SELECT time_bucket(INTERVAL '15 min', time) AS bucket,
    bms.iot_thing,
    bms.iot_device,
    bms.name,
    bms.string,
    bms.module,
    bms.cell,
    avg(bms.value_real) AS value_real,
    last(bms.value_str, time) AS value_str
FROM public.bms
GROUP BY bucket, iot_thing, iot_device, name, string, module, cell;✨Editor’s Note:Here are three reasons you should upgrade to the new version of continuous aggregates.RoadmapWe are very proud and honored to recently have won the competitive call from theEIC Accelerator, led by the European Innovation Council! We’re now working towards industrializing and scaling our battery cloud technology.Advice & ResourcesI believe that getting started with a hands-on example is always a good way to evaluate something. You can try Timescale for free with a demo dataset to get acquainted with the service.So I recommend quickly spinning up a first TimescaleDBinstance and having fun playing around with the service.Also,there is very extensive documentation and tons of examples and tutorialsavailable on the website, which helps you quickly master Timescale! So make sure to have a look at the website.Anything else you'd like to add?Thanks to Timescale for this opportunity to share our story and our mission at Octave!Want to read more developer success stories?Sign up for our newsletterfor more Developer Q&As, technical articles, tips, and tutorials to do more with your data—delivered straight to your inbox twice a month.We’d like to thank Nicolas and all the folks at Octave for sharing how they’re leveraging continuous aggregates and Timescale’s compression powers to handle their millions of battery cell data points daily.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/high-compression-ratio-and-speedy-queries-on-historical-data-while-revolutionizing-the-battery-market/
2022-04-22T13:05:15.000Z,How Edeva Uses Continuous Aggregations and IoT to Build Smarter Cities,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, John Eskilsson, software architect at Edeva, shares how his team collects huge amounts of data (mainly) from IoT devices to help build safer, smarter cities and leverages continuous aggregations for lightning-fast dashboards.Founded in 2009 in Linköping,Edevais a Swedish company that creates powerful solutions for smart cities. It offers managed services and complete systems, including hardware and software platforms.As the creators of the dynamic speed bumpActibumpand the smart city platformEdevaLive, the Edeva team works mainly for municipal, regional, and national road administrations, toll stations, environmental agencies, and law enforcement agencies.The team also solves many other problems, from obtaining large amounts of environmental data for decision-making to developing a screening scale to help law enforcement agencies assess vehicle overloading. The latter, for instance, decreased the amount of time needed to control each vehicle, speeding up traffic checks and allowing law enforcement agencies to control more vehicles.About the TeamTheteam at Edevais a small but impactful group of 11 working on everything from creating hardware IoT devices to analyzing time-series data and making it accessible to customers—and, sometimes—the public.As a software architect, I am in charge of building the best possible solution to receive, store, analyze, visualize, and share the customers’ event data. Our team then comes together to create solutions that work and that the customer actually wants.About the ProjectEdeva has created a dynamic speed bump calledActibumpand the smart city platformEdevaLive.The Actibump has been used in Sweden since 2010. Speeding vehicles activate a hatch in the road that lowers a few centimeters, creating an inverted speed bump. Providing good accessibility for public transportation, such as buses and emergency vehicles, the Actibump still ensures a safe speed for pedestrians and other vulnerable road users. It is also an environmentally friendly solution, helping decrease noise and emissions.The Actibump can be combined with the EdevaLive system, delivering valuable remote monitoring services and statistics to Edeva’s customers.Most of the data we collect is based on IoT devices:Traffic flow data: The Actibump measures the speed of oncoming traffic to decide if it needs to activate the speed bump or not. We capture radar data, among others, and send an event to our smart city platform EdevaLive. The data treats the oncoming traffic as a flow rather than a single vehicle to create the smoothest possible traffic flow.Vehicle classification data (weigh-in-motion): Actibump can be configured with weigh-in-motion. This means that the lid of the speed bump is equipped with a very sensitive high sampling scale. The scale records several weight measurements when the vehicle passes over the speed bump. This way, it can detect how many axles a vehicle has and classify the type of vehicle. At the same time, it fires off one event for each axle with the scale fingerprint so we can analyze if the weight measurements are correct.Vehicle classification data (radar): If we want to classify vehicles in places where we do not yet have an Actibump installed, we can introduce a radar that can classify vehicle types. A roadside server controls the radar, gathers its data, and pushes it to EdevaLive.Bike and pedestrian data: We use cameras installed above a pedestrian and cycle path. The camera can detect and count pedestrians and bicycles passing in both directions. We push this data to EdevaLive for analysis.Number plate data:We can use a camera to detect the number plate of a vehicle. This way, we can control devices like gates to open automatically. It can also be used to look up the amount of electric vs. petrol or diesel vehicles passing the camera or determine if a specific vehicle exceeds the cargo weight limit.Gyroscopic data: We offer a gyroscopic sensor that can gather data for acceleration in all different planes. This device generates a lot of data that can be uploaded to EdevaLive in batches or as a stream (if the vehicle has an Internet connection). This data is GPS-tagged and can be used to calculate jerk to provide indications on working conditions to a bus driver, for instance. The data can also be used to calculate the wear and tear of vehicles and many other things.Environmental data: It is important to monitor environmental data in a smart city platform. This is why we use small portable devices that can measure the occurrence of different particle sizes in the air. It can also measure CO2 and other gasses. On top of that, it measures the usual things like temperature, wind speed, etc. All this data is pushed to EdevaLive.Alarm data: Our IoT devices and roadside servers can send alarm information if a sensor or other parts malfunction. All this data comes to EdevaLive in the same way as a regular IoT event, but these events are only used internally so that we can react as quickly as possible if there is a problem.Status data: If the alarm data detects anomalies, the status data just reports on the status of the server or IoT device. The devices run self-checks and report statistical data, like disk utilization, temperature, and load. This is also just for internal use to spot trends or troubleshoot in case any problems arise. For instance, it is incredibly useful to correlate CPU load with the version number of firmware or other software versions.Administrative data: This is where the power of SQL and time-series data really shines. Let’s say we added a new device, and it has a configuration object that is persistent in a regular table in Timescale. This object keeps some metadata, such as the date it was added to the system or the device's display name. This way, we can use a join easily to pick up metadata about the device and, at the same time, get time-series data for the events that are coming in. There is only one database connection to handle and one query to run.Choosing (and Using!) TimescaleDBWe realized we needed a time-series database a few years ago when we started storing our data in MySQL. At the time, we made a move to MongoDB, and it worked well for us but required quite a bit of administration and was harder to onboard other developers.I looked at InfluxDB but never considered it in the end because it was yet another system to learn, and we had learned that lesson with MongoDB.✨Editor’s Note:For more comparisons and benchmarks,see how TimescaleDB compares to InfluxDB, MongoDB, AWS Timestream, vanilla PostgreSQL, and other time-series database alternatives on various vectors, from performance and ecosystem to query language and beyond.Learning from this journey,  I looked for a solution that plugged the gaps the previous systems couldn’t. That is when I found Timescale and discovered that there was a hosted solution.We are a small team that creates software with a big impact, and this means that we don’t really have time to put a lot of effort into tweaking and administering every single tool we use, but we still like to have control.""With Timescale, our developers immediately knew how to use the product because most of them already knew SQL""Also, since TimescaleDB is basically PostgreSQL with time-series functionality on steroids, it is much easier to onboard new developers if needed. With Timescale, our developers immediately knew how to use the product because most of them already knew SQL.Edeva uses TimescaleDB as the main database in our smart city system. Our clients can control their IoT devices (like the Actibump from EdevaLive) and—as part of that system—see the data that has been captured and quickly get an overview of trends and historical data. We offer many graphs that show data in different time spans, like day, week, month, and years. To get this to render really fast, we use continuous aggregations.""Timescale is basically PostgreSQL with time-series functionality on steroids, it is much easier to onboard new developers if needed""✨Editor’s Note:Learn how you can use continuous aggregates for distributed hypertables.Current Deployment and Future PlansOne of the TimescaleDB features that has had the most impact on our work is continuous aggregations. It changed our dashboards from sluggish to lightning fast. If we are building functionality to make data available for customers, we always aggregate it first to speed up the queries and take the load off the database. It used to take minutes to run some long-term data queries. Now, almost all queries for long-term data are subsecond.For example, we always struggled with showing the 85th percentile of speed over time. To get accurate percentile data, you had to calculate it based on the raw data instead of aggregating it. If you had 200 million events in a hypertable and wanted several years’ data for a specific sensor, it could take you a long time to deliver—users don’t want to wait that long.""It changed our dashboards from sluggish to lightning fast""Now that Timescale introducedpercentile_aggandapprox_percentile, we can actually query continuous aggregations andget reasonably accurate percentile valueswithout querying raw data.✨Editor’s Note:Percentile approximations can be more useful for large time-series data sets than averages. Read how their work in this blog post.Note that “vehicles” is a hypertable where actibump_id is the ID of the dynamic speed bump containing several hundred million records.This is how we build the continuous aggregate:CREATE MATERIALIZED VIEW view1
 WITH (timescaledb.continuous) AS
 SELECT actibump_id,
 timescaledb_experimental.time_bucket_ng(INTERVAL '1 month', time, 'UTC') AS bucket,
 percentile_agg(vehicle_speed_initial) AS percentile_agg
FROM vehicles
GROUP BY actibump_id, bucketAnd this is the query that fetches the data for the graph:SELECT TIMESCALEDB_EXPERIMENTAL.TIME_BUCKET_NG(INTERVAL '1 month', bucket) AS date,
actibump_id,
APPROX_PERCENTILE(0.85, ROLLUP(PERCENTILE_AGG)) AS p85,
MAX(signpost_speed_max)
FROM vehicles_summary_1_month
WHERE actibump_id in ('16060022')
AND bucket >= '2021-01-30 23:00:00'
AND bucket <= '2022-04-08 21:59:59'
GROUP BY date, actibump_id
ORDER BY date ASCHere is an example of the graph:At the moment, we use PHP and Yii 2 to deploy TimescaleDB. We connect to TimescaleDB with Qlik Sense for business analytics. In Qlik Sense, you can easily connect to TimescaleDB using the PostgreSQL integration.It is especially convenient to be able to connect to the continuous aggregations for long-term data without overloading the system with too much raw data. We often use Qlik Sense to rapidly prototype graphs that we later add to EdevaLive.Advice and ResourcesThe next step for us is to come up with a good way of reducing the amount of raw data we store in TimescaleDB. We are looking at how we can integrate it with a data lake. Apart from that, we are really excited to start building even more graphs and map applications.If you are planning to store time-series data, Timescale is the way to go. It makes it easy to get started because it is “just” SQL, and at the same time, you get the important features needed to work with time-series data. I recommend you have a look, especially at continuous aggregations.Think about the whole lifecycle when you start. Will your use cases allow you to use features like compression, or do you need to think about how to store long-term data outside of TimescaleDB to make it affordable right from the start? You can always work around things as you go along, but it is good to have a plan for this before you go live.💻If you want to learn more about how Edeva handles time-series data with Actibump and EdevaLive, the team hostsvirtual biweekly webinars, or you can alsorequest a demo.We’d like to thank John and all the folks from Edeva for sharing their story. We are amazed to see how their work truly impacts the way people live and enjoy their city with a little help from time-series data.🙌We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-edeva-uses-continuous-aggregations-and-iot-to-build-smarter-cities/
2023-05-18T16:52:53.000Z,Visualizing IoT Data at Scale With Hopara and TimescaleDB,"IntroductionI have been involved in building DBMSs for 50 years. During that time, I have been the main architect behind PostgreSQL (the foundation of Timescale), Vertica, VoltDB, and Paradigm4. Recently, I have been studying user-facing problems, most notably, “How do users derive information from the massive amount of data we are collecting?”Sometimes users know exactly what they are interested in, and a conventional dashboard (such as Tableau or Spotfire) can help them find it. Often, however, the real question behind a query a user wants to run is, “Tell me something interesting?” I.e., “Show me an actionable insight.” To provide these meaningful data insights, a sophisticated visualization system is needed to complement more traditional analytics systems.I have long been a fan of Google Maps, which allows you to go from a picture of the Earth to the plot map on your street in 21 clicks. This is an exemplar of a “detail on demand” system, often called a “pan-zoom” interface. A big advantage of such systems is that no user manual is required since the interface is so intuitive. Unfortunately, Google Maps only works for geographic data. So what to do if you have floor plans, 3D models, scatter plots, or the myriad of other representations that users want to see?Hoparacan be thought of as “Google Maps on steroids.” It will produce pan-zoom displays for any kind of data. It is especially applicable for real-time monitoring applications, often from IoT data collection or asset tracking of sensor-tagged devices.Hopara is almost three years old, headquartered in Boston, and employs 12 people.Figure 1. Example of a Hopara app in the lab space (powered by Timescale)In this post, I will walk you through a Hopara monitoring application powered by TimescaleDB. It shows the benefit of Hopara visualization aided by a traditional analytics dashboard. This application reports vibration issues in sensor-tagged machines, and the real-time vibration data is stored in a Timescale database for effective real-time querying.The Problem: Monitoring (Lots of) Sensor Data in Real TimeLet’s consider a French company that operates 58 factories in Brazil, manufacturing construction materials (think pipes, glue, and nails). This company is referred to as FC in the rest of this post.FC is in the process of installing about 50K sensors from a Brazilian vendor, IBBX. These sensors primarily report vibration, typically at 100 msec intervals. Excessive vibration is often an early warning of machine failure or the need for urgent service. Hence, real-time monitoring is required for 50K time series of vibration data.Unlike some applications that can aggregate data in the network from sensor to server, FC requires that details be available on all sensors so abnormal events can be reported on individual machines.These abnormal events can include the following:Latest reading over a thresholdFive sequential readings over five minutes above a second thresholdOne reading per minute over a third threshold for 10 minutesA reading over a fourth threshold more than once a day for a monthIn addition, FC wants to filter machines by location (e.g., only those plants in São Paulo) and by time (e.g., report only weekend vibrations).A time-series database is the optimal solution for storing the 50K time-series sensor data for a long period of time. Although this database is likely measured in GB, it is easy to imagine much larger applications.In the rest of this blog post, I first talk about the FC reporting architecture and their future requirements. Then, I discuss the requirements for the database, and why Hopara ended up using Timescale.From Data to Insights: Building Actionable VisualizationsFC runs a sophisticated analytics application provided by IBBX. It monitors vibrations and employs machine learning-based predictive analytics to forecast needed repair events. The IBBX dashboard for FC is shown in Figure 2. It shows “the details” using typical dashboard technology.  This is very useful for initiating corrective action when required.Figure 2. FC/IBBX dashboard using time-series sensor data in real timeHowever, this dashboard does not show “the big picture” desired by the FC personnel. Hence, IBBX recently partnered with Hopara to produce a “detail-on-demand” system for the same data. Figure 3 shows FC factories on a map of Brazil color-coded with their facility health.Figure 3. The Big Picture: FC factories on a map of Brazil color-coded with their facility healthNotice that there are two factories with a yellow status. If an FC user wants to “drill into” one of them, the user will see Figure 4. Figures 4 and 5 shows greater detail about one yellow sensor in Figure 3, while Figure 6 shows the time series of sensor readings for the sensor in question over five days. Notice that, in a few clicks, a user can move from an overview to the actual time-series data.FC users appreciate both the analytics dashboard and the Hopara drill-down system. As a result, IBBX and Hopara combined their software into a single system. The takeaway from this example is that there is a need for predictive analytics and insightful visualization. IoT customers should have both kinds of tools in their arsenal.✨Editor's Note:If you want to learn more abouttime-series forecasting, its applications, and popular techniques, check out this blog post.Figure 4. The factory in questionWe now turn our attention to response time. It is accepted pragma that the response time for a command to a visualization system must be less than 500 msec. Since it takes some time to render the screen, the actual time to fetch the data from a storage engine must be less than this number. The next section discusses DBMS performance considerations in more detail.Behind The Scenes: Powering Real-Time Visualizations Using TimescaleFigure 5. More real-time detailsFigure 6. The actual dataAs mentioned previously, these real-time views are powered by TimescaleDB. TimescaleDB is built on top of PostgreSQL and extends it with a series of extremely useful capabilities for this use case, such asautomatic partitioning by time,boosted performance for frequently-run queries, andcontinuous aggregates for real-time aggregations.To guarantee a real-time display, Hopara fetches live data from the database for every user command. Otherwise, stale data will be rendered on the screen. The screenshots above come from a real Hopara application interacting with a database. We note in Figure 2 that the alerting condition is the first one mentioned in a previous section (the latest reading over a threshold).In other words, the display is showing a computation based on the most recent values from the various sensors. Specifically, Hopara uses a database with the schema shown in Figure 7, with a Readings table with all the raw data. This is connected to a Sensor table with the characteristics of each individual sensor.Figure 7. The database schemaThen, the display in Figure 2 requires fetching the most recent reading for each sensor. This can be produced by running the following two-step PostgreSQL query:–— get the latest reading timestamp + sensor_id
SELECT max(timestamp) as timestamp, sensor_id
FROM readings
GROUP BY sensor_id


—— get the reading value
SELECT l.timestamp, l.sensor_id, r.value FROM latest l
INNER JOIN readings r ON r.sensor_id = l.sensor_id AND r.timestamp = l.timestampThis query, while easy to understand, is not efficient. The following representation leverages PostgreSQLdistinct onand Timescaleskip scanto perform the query faster, so it is a preferred alternative.SELECT DISTINCT ON (sensor_id) * 
FROM readings
WHERE timestamp > now() - INTERVAL '24 hours' 
ORDER BY sensor_id, timestamp DESC;Note that condition one can be triggered inadvertently, for example, by a worker brushing against the machine. Hence, some combination of conditions 2-4 is a more robust alerting criterion. Unfortunately, these require complex real-time aggregation, which is not present in PostgreSQL. As a result, Hopara switched to TimescaleDB, which extends PostgreSQL with these capabilities (through its continuous aggregate capabilities).Turn now to Figure 6, which displays five days of data for three sensors. Since each sensor is reporting every 100 msec, there are around 4.5M observations in a five-day window. Obviously, this level of granularity is inappropriate for the display presented. In addition, there is no way to produce the display within the required response time. Hence, Hopara aggregates the raw data into two-minute averages using Timescale’scontinuous aggregates(Figure 5 displays these averages).Timescale’shypertablesautomatically partition big tables, so it’s easier for the query planner to localize time-series data. This greatly accelerates queries such as the one required to produce Figure 5, another reason for the switch from PostgreSQL to TimescaleDB.As a result, Hopara and TimescaleDB are powerful tools for IoT applications of the sort discussed in this blog post.We want to thank Professor Michael Stonebraker and the team at Hopara for sharing their story on how they are providing meaningful visualization solutions for their customers’ sensor data. As PostgreSQL lovers and enthusiasts, we are incredibly grateful and proud to see Professor Stonebraker using TimescaleDB to provide real-time insights to Hopara’s customers.If you want to learn more about Timescale and see how we handle time-series data, events, and analytics,read our Developer Q&As. These are real stories from real developers working in the real world. And if you want to try our cloud solution, sign up for a 30-day free trial. You will have access to all its unique features, from continuous aggregations to advanced analytical functions (and you don’t even need a credit card!).About the AuthorMichael Stonebrakeris a pioneer of database research and technology. He joined the University of California, Berkeley, as an assistant professor in 1971 and taught in the computer science and EECS departments for 29 years.While at Berkeley, he developed prototypes for the INGRES relational data management system, the object-relational DBMS, POSTGRES, and the federated system, Mariposa. He is the founder of three successful Silicon Valley startups whose objective was to commercialize these prototypes.Mike is the author of scores of research papers on database technology, operating systems, and the architecture of system software services. He was awarded the ACM System Software Award in 1992 (for INGRES) and the Turing Award in 2015.He was elected to the National Academy of Engineering and is presently an adjunct professor of computer science at MIT’s Computer Science and AI Laboratory (CSAIL.)Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/visualizing-iot-data-at-scale-with-hopara-and-timescaledb/
2022-03-15T15:28:55.000Z,Automated Mocking Using API Traffic: Speedscale's Story,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Ken Ahrens, co-founder and CEO of Speedscale, joins us to share how they modernize and automate testing practices for cloud infrastructure by providing API traffic data going into and out various microservices. Thanks to TimescaleDB, Speedscale´s UI loads quickly and provides the ability to drill deep down into high-fidelity data.Speedscaleis one of the first commercial technologies utilizingactual API trafficin order to generate tests and mocks. Speedscale helps Kubernetes engineering teams validate how new code will perform under production-like workload conditions.Speedscale service mapSpeedscale provides unparalleled visibility, collects and replays API traffic, introduces chaos, and measures the golden signals of latency, throughput, saturation, and errors before the code is released.Screenshot of Speedscale UISpeedscale Traffic Replayis a modern load, integration, and chaos testing framework -- an alternative to legacy scripting tools that can take days or weeks to run and do not scale well for modern architectures.If your organization provides a SaaS product that is critical for revenue, or your team is responsible for performant infrastructure, Speedscale is for you. Speedscale enables engineering leaders to generate quality automation quickly without the need for manual scripting. Since actual API calls (not scripts) are the primary ingredient, tests and mocks can be built and regenerated quickly to keep pace with the speed of changing business requirements.As microservices become more separated logically, they are highly dependent on each other to deliver the expected functionality. This means performance problems become distributed across multiple services and can be difficult to trace.  Multiple contributing factors affect the state of an application in microservices and Kubernetes environments. A testing harness that closely mirrors the production setup and incoming traffic has become a requirement for highly distributed and containerized environments.By leveraging traffic to automatically generate sophisticated mocks, engineers and testers are granted the ability to isolate and contract/performance test smaller components in the context of a tightly coupled architecture.  This superpower enables rapid testing iterations. Moreover, without the need for scripting, testing can finally move as fast as development.About the foundersSpeedscale’s leadership team comes from companies like New Relic, Observe Inc, Wily Introscope (bought by CA Technologies), and iTKO (bought by CA Technologies).My name isKen Ahrens. I am co-founder and CEO of Speedcale. Much of my career has been focused on helping companies develop and manage complex web applications. I previously ran North America teams for New Relic and CA/Broadcom. Previous startups included Pentaho (acquired by Hitachi), ITKO (acquired by CA/Broadcom), and ILC (acquired by General Dynamics). My first foray into programming started with a brand new language called Java at Georgia Tech and has grown into a lifetime interest.Matthew LeRay, co-founder and CTO at Speedscale, has invested the past 20 years in improving the performance of applications across multiple generations of technology. Previously, he was head of product at Observe, SVP at CA Technologies (acquired by Broadcom), and engineering leader at ILC (acquired by General Dynamics). He is an alumnus of Georgia Tech in both Computer Engineering and Business. His first love is debugging Golang code but he occasionally takes a break to craft hand-carved guitars.Nate Lee, co-founder and VP of Sales & Marketing, has served a variety of roles within the software industry. Most recently, he was in enterprise sales for the digital transformation consultancy Contino (acquired by Cognizant). Prior to Contino, he served as Product Manager at CA Technologies, by way of iTKO where he was a Presales Engineer for 6 years. Before iTKO, he spent time as a support leader at IBM Internet Security Systems, and engineer at ILC (acquired by General Dynamics). He graduated from Georgia Tech with an MBA in Technology, and a BS in Computer Science.  You’ll most likely find him outdoors on 2 wheels when he’s not innovating with his Speedscale buddies.As a small, nimble startup, our normal workweek is comprised mostly of helping customers scope their use cases and deciding where to start with our testing framework (recording traffic, generating traffic replay “scenarios” comprised of tests and mocks of auto-identified dependencies). We are an engineering startup with a heavy emphasis on Kubernetes and Golang. We also are improving the protocol and version support of both our Enterprise Kubernetes version and Desktop version called theSpeedscale CLI.About the projectWe noticed the application of outdated testing practices to modern cloud infrastructure (eg. UI testing, manual testing, API test tools with no mocking). As the number of connections in distributed, containerized applications grew, the need for quality automation increases exponentially. As a result, the popularity ofcanary releases and blue green deploymentshave risen, but we believe this is due to a lack of robust quality automation alternatives.Speedscale uses a transparent proxy to capture API transaction data from which to model robust API tests and realistic mocks of backend dependencies. Our traffic replay framework iscapable of showing engineering teamscomplex headers, authentication tokens, message bodies, and associated metadata (with sensitive data redaction when necessary). In addition, the platform is able toautomatically identify backend dependenciesthat are needed to operate your service, allowing new developers on old services to get up to speed quickly. This data is streamed to AWS S3 and filtered for test and mock scenario creation. These scenarios can then be replayed as part of validation test suites and integrated into CI or GitOps workflows.Our customers need to be able to understand the API traffic going into and out of their various microservices over time. They want to see the sequence of API calls as well as the trend of the overall volume of calls.Data is ingested by our platform into our cloud data warehouse. As new data arrives, we determine the index where that API call can be found and write the index to TimescaleDB. Then we can use the data from TimescaleDB to find the original value. Because the indexes are much smaller than the original data, we are able to calculate aggregates on the fly and plot them in our user interface. The Traffic Viewer graph shows inbound and outbound calls, backend dependencies, and an “infinite scroll” list of traffic. All of these components are powered by TimescaleDB queries.Speedscale Traffic ViewerChoosing (and using!) TimescaleDBWe knew right from the start that we have a time-series problem. There was always new data flowing in, and users wanted to focus on data from certain periods of time, they didn't just want all the data presented to them. We decided to use a time-series database to store the data.We wanted a technology that could run inside Kubernetes, is easy to operate (we are a startup after all) and scale for our needs. We initially implementedElasticsearchand exposed the data throughKibana. It let us quickly prototype the use cases and worked great for lower volumes of data. But it scaled poorly for our use case and we had very little control over the look and feel of the UI. Then we evaluated TimescaleDB, Influx, Prometheus and Graphite.We selected TimescaleDB because we were already using PostgreSQL as part of our technology stack, and also the paper evaluation looked like TimescaleDB would scale well at our load ranges.✨Editor’s Note:For more comparisons and benchmarks, seehow TimescaleDB compares to InfluxDB, MongoDB, AWS Timestream, vanilla PostgreSQL, and other time-series database alternativeson various vectors, from performance and ecosystem to query language and beyond.We useSQiurreLto issue SQL queries like this one that powers the inbound throughput graph.rrPairsQuery := sq.Select(
		fmt.Sprintf(""time_bucket(INTERVAL '%s', time) AS bucket"", durationToSQLInterval(interval)),
		""is_inbound"",
		""COUNT(id)"").
		From(rrPairsTableName).
		GroupBy(""is_inbound"", ""bucket"").
		OrderBy(""bucket"")We deploy TimescaleDB on theKubernetes operatorviaFlux. Our core services are currently written in Golang which we use to connect TimescaleDB with microservices inside Kubernetes as well as AWS Lambda.Currently, our architecture that touches TimescaleDB looks like this:Current Speedscale architecture diagramAfter implementing TimescaleDB, our AWS cloud costs went down about 35% because it is cheaper to run than the AWS OpenSearch we used before. In addition, the query performance improved dramatically, a majority of queries take under 100ms to complete.Advice & resourcesHaving aKubernetes operatorwas a big help for us because it was proof that this was built for our architecture.We’ve made a version of our traffic capture capability available as a free CLI which you can find here:https://github.com/speedscale/speedscale-cliWe’d like to thank Ken and all folks from Speedscale for sharing their story. We applaud your efforts to modernize and automate testing practices for modern cloud infrastructure.🙌We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Lucie Šimečková), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/automated-mocking-using-api-traffic-speedscales-story/
2020-08-07T23:16:00.000Z,How I Power a (Successful) Crypto Trading Bot With TimescaleDB,"This is an installment of our “Community Member Spotlight” series, where we invite TimescaleDB community members to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Felipe Queis, a senior full-stack engineer for a Brazilian government traffic institution, joins us to share how he uses TimescaleDB to power his crypto trading bot – and how his side project influenced his team’s decision to adopt TimescaleDB for their work.My first experience with crypto wasn’t under very good circumstances: a friend who takes care of several servers at his job was infected with ransomware – and this malware was demanding he pay the ransom amount in a cryptocurrency called Monero (XMR).After this not-so-friendly introduction, I started to study how the technology behind cryptocurrencies works, and I fell in love with it. I was already interested in the stock market, so I joined the familiar (stock market) with the novel (crypto). To test the knowledge I’d learned from my stock market books, I started creating a simpleMoving Average Convergence Divergence (MACD)crossover bot.This worked for a while, but I quickly realized that I should - and could - make the bot a lot better.Now, the project that I started as a hobby has a capital management system, a combination of technical indicators, and sentiment analysis powered by machine learning. Between 10 March 2020 and 10 July 2020, my bot resulted in asuccess rateof 61.5%,profit factorof 1.89, and cumulative gross result of approximately 487% (you can see a copy of all of my trades during this period inthis Google Sheet report).About meI'm 29 years old, and I’ve worked in a traffic governmental institution in São Paulo, Brazil (where I live too) as an senior full-stack developer since 2012.In my day job, my main task at the moment is processing and storing the stream of information from Object Character Recognition (OCR)-equipped speed cameras that capture data from thousands of vehicles as they travel our state highways. Our data stack uses technologies like Java, Node.js, Kafka, and TimescaleDB.(For reference, I started using TimescaleDB for my hobby project, and, after experiencing its performance and scale with my bot, I proposed we use it at my organization. We’ve found that it brings together the best of both worlds: time-series in a SQL databaseandopen source).I started to develop my crypto trading bot in mid- 2017, about six months after my first encounter with the crypto ecosystem – and I’ve continued working on it in my spare time for the last two and a half years.Editor’s Note: Felipe recently hosted aReddit AMA (Ask Me Anything)to share how he’s finally “perfected” his model, plus his experiences and advice for aspiring crypto developers and traders.About the projectI needed a bot that gave me a high-performance, scalable way to calculate technical indicators and process sentiment data in real-time.To do everything I need in terms of my technical indicators calculation, I collectcandlestick chartdata and market depth via an always-up websocket connection that tracks every Bitcoin market on theBinance exchange(~215 in total, 182 being tradeable, at this moment).The machine learning sentiment analysis started as a simple experiment to see if external news affected the market. For example: if a famous person in the crypto ecosystem tweeted that a big exchange was hacked, the price will probably fall and affect the whole market. Likewise, very good news should impact the price in a positive way. I calculated sentiment analysis scores in real-time, as soon as new data was ingested from sources like Twitter, Reddit, RSS feeds, and etc. Then, using these scores, I could determine market conditions at the moment.Now, I combine these two components with a weighted average, 60% technical indicators and 40% sentiment analysis.Felipe's TradingBot dashboard, where he tracks all ongoing trades and resultsQuick breakdown of Felipe’s results and success rates week-over-week (for the period of 10 March 2020 - 10 July 2020)Using TimescaleDBAt the beginning, I tried to save the collected data in simple files, but quickly realized that wasn’t a good way to store and process this data. I started looking for an alternative: a performant database.I went through several databases, and all of them always lacked something I wound up needing to continue my project. I tried MongoDB, InfluxDB, and Druid, but none of them 100% met my needs.Of the databases I tried,InfluxDB was a good option; however, every query that I tried to run was painful, due to their own query language (InfluxQL).As soon as my series started to grow exponentially to higher levels, the server didn't have enough memory to handle them all in real-time. This is because the currentInfluxDB TSMstorage engine requires more and more allocated memory for each series. I have a large number of unique metrics, so the process ran out of available memory quickly.I handle somewhat large amounts of data every day, especially on days with many market movements.On average, I’m ingesting around 20k records/market, or 3.6 million total records, per day (20k*182 markets).This is where TimescaleDB started to shine for me. It gave me fast real-time aggregations, built-in time-series functions, high ingestion rates – and it didn’t require elevated memory usage to do all of this.Editor’s Note:For more about how Flux compares to SQL and deciding which one is right for you,see our blog post exploring the strengths and weaknesses of each.To learn more about how TimescaleDB real-time aggregations work (as well as how they compare to vanilla PostgreSQL), seethis blog post and mini-tutorial.In addition to this raw market data, a common use case for me is to analyze the data in different time frames (e.g., 1min, 5min, 1hr, etc.). I maintain these records in a pre-computed aggregate to increase my query performance and allow me to make faster decisions about whether or not to enter a position.For example, here’s a simple query that I use a lot to follow the performance of my trades on a daily or weekly basis (daily in this case):SELECT time_group, total_trades, positive_trades, 
	negative_trades,
	ROUND(100 * (positive_trades / total_trades), 2) AS success_rate, profit as gross_profit,
    ROUND((profit - (total_trades * 0.15)), 2) AS net_profit
FROM (
	SELECT time_bucket('1 day', buy_at::TIMESTAMP)::DATE AS time_group, COUNT(*) AS total_trades, 
		SUM(CASE WHEN profit >  0 THEN 1 ELSE 0 END)::NUMERIC AS positive_trades, 
		SUM(CASE WHEN profit <= 0 THEN 1 ELSE 0 END)::NUMERIC AS negative_trades,
		ROUND(SUM(profit), 2) AS profit 
	FROM trade
	GROUP BY time_group ORDER BY time_group 
) T ORDER BY time_groupAnd, I often use this function tomeasure market volatility, decomposing the range of a market pair in a period:CREATE OR REPLACE FUNCTION tr(_symbol TEXT, _till INTERVAL)
	RETURNS TABLE(date TIMESTAMP WITHOUT TIME ZONE, result NUMERIC(9,8), percent NUMERIC(9,8)) LANGUAGE plpgsql AS $$ DECLARE BEGIN

RETURN QUERY 
	WITH candlestick AS ( SELECT * FROM candlestick c WHERE c.symbol = _symbol AND c.time > NOW() - _till )
	SELECT d.time, (GREATEST(a, b, c)) :: NUMERIC(9,8) as result, (GREATEST(a, b, c) / d.close) :: NUMERIC(9,8) as percent FROM ( 
		SELECT today.time, today.close, today.high - today.low as a,
      		COALESCE(ABS(today.high - yesterday.close), 0) b,
      		COALESCE(ABS(today.low - yesterday.close), 0) c FROM candlestick today
      	LEFT JOIN LATERAL ( 
			  SELECT yesterday.close FROM candlestick yesterday WHERE yesterday.time < today.time ORDER BY yesterday.time DESC LIMIT 1 
		) yesterday ON TRUE
    WHERE today.time > NOW() - _till) d;
END; $$;

CREATE OR REPLACE FUNCTION atr(_interval INT, _symbol TEXT, _till INTERVAL)
	RETURNS TABLE(date TIMESTAMP WITHOUT TIME ZONE, result NUMERIC(9,8), percent NUMERIC(9,8)) LANGUAGE plpgsql AS $$ DECLARE BEGIN
	
RETURN QUERY
	WITH true_range AS ( SELECT * FROM tr(_symbol, _till) )
	SELECT tr.date, avg.sma result, avg.sma_percent percent FROM true_range tr
	INNER JOIN LATERAL ( SELECT avg(lat.result) sma, avg(lat.percent) sma_percent
		FROM (
			   SELECT * FROM true_range inr
			   WHERE inr.date <= tr.date
			   ORDER BY inr.date DESC
			   LIMIT _interval
			 ) lat
		) avg ON TRUE
  WHERE tr.date > NOW() - _till ORDER BY tr.date;
END; $$;

SELECT * FROM atr(14, 'BNBBTC', '4 HOURS') ORDER BY dateWith TimescaleDB, my query response time is in the milliseconds, even with this huge amount of data.Editor’s Note: To learn more about how TimescaleDB works with cryptocurrency and practice running your own analysis,check out our step-by-step tutorial. We used these instructions toanalyze 4100+ cryptocurrencies, see historical trends, and answer questions.Current Deployment & Future PlansTo develop my bot and all its capabilities, I used Node.js as my main programming language and various libraries:Coteto communicate between all my modules without overengineering,TensorFlowto train and deploy all my machine learning models, andtulindfor technical indicator calculation, as well as various others.I modified some to meet my needs and created some from scratch, including a candlestick recognition pattern, a level calculator for support/resistance, and Fibonacci retracement.Current TradingBot architecture + breakdown of various Node.js librariesToday, I have a total of 55 markets (which are re-evaluated every month, based on trade simulation performance) that trade simultaneously 24/7; when all my strategy conditions are met, a trade is automatically opened. The bot respects my capital management system, which is basically to limit myself to 10 opened positions and only use 10% of the available capital at a given time. To keep track of the results of an open trade, I use dynamicTrailing Stop LossandTrailing Take Profit.The process of re-evaluating a market requires a second instance of my bot that runs in the background and uses my main strategy to simulate trades in all Bitcoin markets. When it detects that a market is doing well, based on the metrics I track, that market enters the main bot instance and starts live trading. The same applies to those that are performing poorly; as soon as the main instance of my bot detects things are going badly, the market is removed from the main instance and the second instance begins tracking it. If it improves, it's added back in.As every developer likely knows all too well, the process of building a software is to always improve it. Right now, I’m trying to improve my capital management system usingKelly Criterion, assuggested by a userin my Reddit post (thanks, btw :)).Getting started advice & resourcesFor my use case, I’ve found TimescaleDB is a powerful and solid choice: it’s fast with reliable ingest rates, efficiently stores and compresses a huge dataset in a way that’s manageable and cost-effective, and gives me real-time aggregation functionality.TheTimescale website,""using TimescaleDB"" core documentation, andthis blog post about about managing and processing huge time-series datasetsis all pretty easy to understand and follow – and the TimescaleDB team is responsive and helpful (and they always show up in community discussions, likemine on Reddit).It’s been easy and straightforward to scale, without adding any new technologies to the stack. And, as an SQL user, TimescaleDB adds very little maintenance overhead, especially compared to learning or maintaining a new database or language.We’d like to thank Felipe for sharing his story, as well as for his work to evangelize the power of time-series data to developers everywhere. His success with this project is an amazing example of how we can use data to fuel real-world decisions – and we congratulate him on his success 🎉.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@lacey butler), and we’ll go from there.Additionally, if you’re looking for more ways to get involved and show your expertise, check out theTimescale Heroesprogram.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-i-power-a-successful-crypto-trading-bot-with-timescaledb/
2023-06-27T16:13:26.000Z,Saving Devs’ Time and Compute Power With Retention Policies: The Story of Crypto Trading Platform Pintu,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Christian Halim, engineering manager atPintu, one of Indonesia’s leading cryptocurrency trading platforms, explains how his team inserts five million data rows into Timescale every 30 minutes using Python and AWS Lambda, queries more than a billion data rows in 0.1 seconds, and saves developers’ time and computational power by using retention policies to delete a billion data rows per day automatically.About the CompanyFounded in 2020,Pintu is one of Indonesia's leading cryptocurrency trading platforms, with over six million app downloads.In the first quarter of 2023, by trading volume, Pintu was in Indonesia's top three largest crypto platforms. It has the largest community base, with over one million users across the community and social media platforms.In Indonesia, Pintu is fully regulated and monitored by Commodity Futures Trading Regulatory Agency, also known as BAPPEBTI.There are several unique value propositions for Pintu users, including:Simple and easy UI/UX for even first-time investors, from instant Know Your Customer (KYC), 24/7 fast on-ramp and off-ramp through all major banks and e-wallets, and start trading from as low as 11,000 Indonesian Rupiah (IDR), less than $1.Fully regulated and compliantunder BAPPEBTI.Pintu Academy: the one-stop educational content platform for users to learn about various blockchains, cryptocurrencies, and the technologies behind them.Multiple crypto products in one platform, including: spot, earn, and staking.About the TeamPintu’s Engineering team has around 100 engineers, including frontend engineers, backend engineers, data scientists/engineers, site reliability engineers, and software development engineers in Test. At the time of this writing, Pintu has around 10 teams.My team is responsible for handling Pintu's transaction records and the users' balance. We mainly use Go and PostgreSQL for our tech stacks, withgRPCas the communication protocol between services. We design and develop new features for Pintu brought by our Product team.About the ProjectWe use Timescale for theaccount value chart, an analytics feature showing users' assets' value history in a specific timeframe.The account value chart is a graphical aid you can use to monitor the value of your assets on Pintu. There are two kinds of account value charts:1.Total asset value chart:it displays the value of all your assets in Pintu in Rupiah (Indonesia's official currency) value. You can access this chart on the main page of the Wallet menu.2.Individual asset value chart:it displays the value of each of your assets in Rupiah and Token values. You can access this chart on each asset's Wallet page.Pintu's UI: Total Wallet Value in Rupiah on the left, and Token Total Asset on the rightThe ProblemWe mainly used PostgreSQL as our database hosted in the cloud, and we thought it would take a lot of effort to implement our account value chart feature using regular PostgreSQL.While brainstorming for solutions, we stumbled upon Timescale and decided to use it—it was actually our VP of Engineering, Albert Widiatmoko, who searched online for “time-series database.” He had already heard about Timescale and followed the company’sGitHubandTwitterprior to the search.Although we had tried MongoDB, we saw some benefits in using Timescale instead of MongoDB to store our price chart data: we don't need to maintain the logic for creating OHLC manually and can useTimescale's hyperfunction instead. Also, Timescale has better query performance and automatic obsolete data deletion [using retention policies].What other reasons led us to Timescale?We felt it was a good fit for our problem since we needed periodic refreshes and aggregate data to display to the user.Most of us were familiar with PostgreSQL already.We query and serve our data from Timescale to users using Go. Aside from Go, we also use Python, which we run in AWS Lambda to insert data into Timescale periodically.We support around 200 cryptocurrencies and three million users at Pintu at the moment. To give you a more detailed picture, our Lambda runs once every 30 minutes. Each time it runs, it inserts about five million rows of data. At a point in time, there are around 1.4 billion rows in the raw table since we usea retention policy to delete obsolete data.✨Editor’s Note:Check out this blog post and learn how tobuild and deploy a serverless time-series application on AWS Lambda and Timescale using SAM CLI.Impact of Using TimescaleTime to market is a crucial factor for us, and Timescale helped us a lot in this aspect since it already supports most of the features we wanted to develop. We could also extend the features later in the future when needed since Timescale supports a variety of aggregates for time-series data.“Our queries are really fast, taking only 100 ms for a table with around 1.4 billion rows”The feature that we developed, the account value chart, was the main topic of discussion in our community when it was first released since customers could share their gainings or losses over a period of time.We used Timescale right from the start for this feature, so we can’t compare Timescale’s performance to vanilla PostgreSQL. However, our queries are really fast, taking only 100 ms for a table with around 1.4 billion rows.Sample query:SELECT namespace, external_id, asset_type_id, balance, balance_idr, at
FROM user_asset_value_chart_1_day
WHERE namespace = $1
	AND external_id = $2
	AND at > $3
	AND asset_type_id = $4
ORDER BY at ASCWe mainly utilizedcontinuous aggregatesto create this feature, which greatly helped us since we didn't have to maintain multiple tables for each timeframe.✨Editor's Note:Continuous aggregates are Timescale's version ofPostgreSQL's incrementally-updated materialized views.Currently, we support several timeframes, and in each timeframe, the interval of stored data is different. For example, in a one-day timeframe, we use 30-minute intervals; in a three-month timeframe, we use 12-hour intervals since the data will be too granular if we use 30-minute intervals instead. This means we would have 3 * 30 * 24 * 2 data per user (30-minute interval) vs. 3 * 30 * 2 data per user (12-hour interval).--Pintu's continuous aggregates

SELECT add_continuous_aggregate_policy('wallet_asset_value_chart_1_day', start_offset => INTERVAL '2h30m', end_offset => INTERVAL '30m', schedule_interval => INTERVAL '30m');


SELECT add_continuous_aggregate_policy('wallet_asset_value_chart_1_week', start_offset => INTERVAL '5h', end_offset => INTERVAL '1h', schedule_interval => INTERVAL '1h');


SELECT add_continuous_aggregate_policy('wallet_asset_value_chart_1_month', start_offset => INTERVAL '20h', end_offset => INTERVAL '4h', schedule_interval => INTERVAL '4h');


SELECT add_continuous_aggregate_policy('wallet_asset_value_chart_3_months', start_offset => INTERVAL '2 days 12h', end_offset => INTERVAL '12h', schedule_interval => INTERVAL '12h');


SELECT add_continuous_aggregate_policy('wallet_asset_value_chart_6_months', start_offset => INTERVAL '5 days', end_offset => INTERVAL '1 day', schedule_interval => INTERVAL '1 day');Our team also saved time using retention policies, so we didn't have to delete obsolete data manually. We should delete around 240 million rows of data daily from the raw table. But since we use retention policies on each continuous aggregate—we have seven continuous aggregates for each timeframe—we should delete approximately 800 million rows of data. This adds up to around one billion deleted data rows per day when the policies run simultaneously.Regarding hyperfunctions, we only used thefirst()hyperfunction in our account value chart feature as we simply needed to get users’ asset values for a specific time.Right now, only a handful of our devs really understand Timescale since the features we developed so far rarely involve Timescale. Still, we are planning to share the knowledge with other engineers. We haven’t really developed other features outside this, but we believe it will be much faster if we utilize the same data as the base.“While we can manually delete obsolete data without Timescale, we saved our developers’ effort and the server's computational power [by creating a retention policy]”Roadmap and Future WorkCurrently, we haven’t any concrete future project utilizing Timescale in mind. However, we have also used Timescale for storing our cryptocurrency price chart, so I believe we’ll keep using Timestable in the foreseeable future since we already have the foundation built.Advice & ResourcesTimescale is very powerful if your data is time series, and you need to have plenty of analytics toward the data. Theirofficial docsare already great, so I really recommend developers learn from there.If there is any advice I can give, I think it’s utilizing cost-saving Timescale features, such ascompressionand retention policies, when possible. We save on storage with a retention policy since we automatically delete obsolete data.While we can manually delete obsolete data without Timescale, we saved our developers’ effort and the server's computational power, although, unfortunately, I don't have the exact data for how much we saved.Anything else you would like to add?Great Customer Support! Tanya and Hardik helped us a lot during our development stage.Facing similar problems while building your application?See our docs to learn how to create a retention policy.Read Messari's use case (and see if you relate).Follow our tutorial on setting up and querying a blockchain dataset.We’d like to thank Christian Halim and the folks at Pintu for sharing their story on accelerating their time to market, saving with compression and retention policies, and providing users with quick insight into their gainings or losses.We’re always keen to feature new community projects and stories on our blog. If you have a project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/saving-devs-time-and-compute-power-with-retention-policies-the-story-of-crypto-trading-platform-pintu/
2022-07-19T13:00:00.000Z,"Using IoT Sensors, TimescaleDB, and Grafana to Control the Temperature of the Nuclear Fusion Experiment at the Max Planck Institute","This is an installment of our “Community Member Spotlight” series, where we invite our customers or users to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, David Bailey, an intern at the Wendelstein 7-X fusion reactor experiment at theMax Planck Institute for Plasma Physics, explains how he’s using TimescaleDB and Grafana to monitor the reliability of the reactor’s heating system. Currently working on the designs of the sensor boards that measure the microwaves feeding the reactor, David needs a tight grasp on his systems and data to prevent the experiment from being aborted due to excessive heating—and is successfully achieving it with a little help from TimescaleDB.About the UserI am David Bailey, and I am currently completing an internship at the Wendelstein 7-X fusion reactor experiment, which is operated by the Max Planck Institute for Plasma Physics in Greifswald, Germany. The experiment aims to help us understand, and someday utilize, the power of fusion for further scientific endeavors and power generation. It does this by using a novel way of containing the hot gasses needed for the experiments and tweaking its design to reach longer and longer experiment runs.David BaileyThe end goal is to reach 30 minutes of uninterrupted containment of plasma heated to millions of degrees. To be able to do this, there is a lot of data to collect and process and a number of problems left to solve—one of which I can help with!About the ProjectMy specific task at the internship is to help with the heating system. Because the experiment is not set up for self-sustaining fusion, where the gasses would heat themselves, we constantly have to feed in energy to keep it hot: 10 megawatts, to be precise!We do this with various microwave sources—similar to scaled-up versions of household microwaves—and a big array of mirrors to guide the energy to the reactor vessel.Schematic of the W7-X microwave heating system and mirror assembly. Credit: Torsten Stange, David Bailey; Max Planck Institute GreifswaldThere are dangers in using these highly powerful microwaves: if dust or a fine droplet of water gets in the way of the energy, an arc can form—a lightning bolt of sorts. If the energy is not turned off fast enough, this arc can do a lot of damage, and it also means that we need to abort the experiment to wait for things to cool off again—not good if you want to run everything reliably for long periods of time!Image of a forming microwave arc, taken at the Max Planck IPP Greifswald experiment siteMy task is to design sensor boards to precisely track the amount of energy coming out of the microwave array. The ultimate goal is to detect changes in the power output on the microsecond scale, so the power can be turned off before the arc fully forms. If done right, we can turn the heating back on without a pause and continue the experiment!One of the key aspects of what I am designing is that it needs to be reliable. If it's not sensitive enough, or too sensitive, or if there are issues with the communication with the rest of the system, it can severely impact the rest of the experiment in a negative way.One of the sensors that David is working onThe only way to ensure something is reliable is through data—a lot of it. A problem might present itself only after hundreds of hours or in subtle ways that are only apparent across days of data, but it can still be relevant and important to know about.​Writing a program to handle this amount of data myself would've been an unnecessary effort. It needs a tool that has the necessary functionality in it already, such as statistical operators, compression, etc., and you can get all of this in time-series databases, such as TimescaleDB!To track that the sensors and system are working as expected, I collect and handle several types of data using TimescaleDB:I am recording the general health metrics of the board in question: temperature, voltage levels, etc. These shouldn't change, but the harsh environment close to the reactor might cause issues, which would be very important to know about!Log messages of the boards themselves to understand what the software was doing.And finally, detection events: every time the board sees something suspicious, it sends a measurement series of this event, about 1,000 samples taken over a millisecond. We can use this during the initial development phase to ensure everything is working correctly. I can use a function generator to send a predetermined reference signal and compare that to what the board tells me it saw. If there are discrepancies, those might point to errors in the soft- or hardware and help me fix them.When the system is deployed, we can use these measurements to refine the detection system, to be able to set it as fast as possible without too many false positives, and gather data about how the actual physical events look to the board.Choosing (and Using!) TimescaleDB✨Editor's Note:See how you can get started with Grafana and TimescaleDB with ourdocsorvideos.I started using Timescale to track data formy self-built smart homeas a small playground for me—that's still running! Aside from that, I only use it for the monitoring system mentioned above, though I hope to motivate others at my research institute to use it! It is also my go-to for other projects, should I need a way to store measurements efficiently.The top factors in my decision to use TimescaleDB were, first, the open-source nature of it. Open-source software can be much more beneficial to a wider range of people, mature faster, generally has more flexible features, and isn't so locked into a specific environment. It's also much more approachable to individuals because of no fussy licensing/digital rights management.Then, the documentation. You can read up on a lot of things that TimescaleDB can do, complete with great examples! It doesn't feel like you're alone and have to figure it out all by yourself—there's rich, easy-to-understand documentation available, and if that doesn't have what you need, the community has been very helpful, too.✨Editor's Note:Need help with TimescaleDB? Join ourSlack CommunityorForumand ask away!And lastly,it's still all just SQL! You don't need to learn another database-specific format, but you can start using it immediately if you've worked with any standard database before! That was helpful because I already knew the basics of SQL, giving me a great starting point. Plus, you can use PostgreSQL's rich relational database system to easily store non-time-series data alongside your measurements.Because the Wendelstein is fairly old, database tools didn’t exist when it started. As such, they decided to write their own tool,called ArchiveDB, and never quite brought it to the modern age. It can only store time-series data in a particular format and has no relational or statistical tools aside from minimum/maximum aggregates, no continuous aggregates, and probably no compression.“In the future, I want to motivate my research institute to install a proper multi-node TimescaleDB cluster. It could bring many much-needed modern features to the table, and distributed hypertables and the matching distributed computing of statistics could be incredibly useful”Storing all my measurement data in it would have been possible, but I would have had to write all of the statistical processing myself. Using Timescale was a major time-saver! I tried other tools, but none of them quite fit what I needed. I didn't want to spend too much time on an unfamiliar tool with sparse documentation.First, I tried the company’s standard database, ArchiveDB, but it didn’t offer much more functionality than simply saving a CSV file. I also tried another PostgreSQL extension,Citus. It didn’t lend itself nicely to my use case because the documentation was not as clear and easy, and it seemed much more tailored to a distributed setup from the start, making trying it out on my local machine a bit tricky. I believe I gaveInfluxDBa try, but writing in PostgreSQL felt more natural. Additionally, it was helpful to have the regular relational database from PostgreSQL and the rich set of aggregate functions.I might have used PostgreSQL without TimescaleDB,or perhaps InfluxDB had it not been for Timescale.The Grafana panels observing the sample measurements. A single event is plotted above for inspection using a scatter plot, while the accuracy of the individual measurements is plotted in a time-series belowOne of the things I am verifying right now is the measurements’ consistency. This means running thousands of known reference signals into the boards I am testing to see if they behave as expected. I could automate this using a query that JOINs a regular table containing the reference data with aTimescaleDB hypertablecontaining the measurements.With a bit of clever indexing, I was able to use the corr(x, y) function to check if things lined up. Thanks to the rich data types of PostgreSQL, I can cache metadata such as this correlation in a JSONB field. This speeds up later analysis of the data and allows me to store all sorts of extra values for individual events.The resulting data I then downsampled usingminto find the worst offenders.WITH missing_bursts AS ( 
  SELECT burst_id FROM adc_burst_meta WHERE NOT metadata ? 'correlation' AND $__timeFilter(time)),
raw_correlations AS (
    	SELECT
    	burst_id,
    	corr(value, reference_value) AS ""correlation""
    	FROM adc_burst_data
    	INNER JOIN missing_bursts USING(burst_id)
    	INNER JOIN adc_burst_reference USING(burst_offset)
    	GROUP BY burst_id
)    
update_statement AS (
     UPDATE adc_burst_meta 
    SET metadata = jsonb_set(metadata, '{correlation}', correlation::text::jsonb) FROM raw_correlations WHERE adc_burst_meta.burst_id = raw_correlations.burst_id
)

SELECT $__timeGroup(time, $__interval) AS “time”, 
   min((metadata->>’correlation’)::numeric)
FROM adc_burst_meta 
WHERE $__timeFilter(time) 
GROUP BY “time”That made even small outliers very easy to spot across hundreds of thousands of samples and made analyzing and tagging data a breeze! This insight is incredibly useful when you want to ensure your system works as intended. I could easily find the worst measurements in my sample set, which allowed me to quickly see if, how, and in what way my system was having issues.Current Deployment and Future Plans​Right now, I deploy TimescaleDB on a small, local computer.Running the Docker image has been very useful in getting a fast and easy setup, too!In the future, I want to motivate my research institute to install a proper multi-node TimescaleDB cluster. It could bring many much-needed modern features to the table, and distributed hypertables and the matching distributed computing of statistics could be incredibly useful! My queries could work orders of magnitude faster than my local machine with little extra effort.For the most part, I interact with TimescaleDB through a simple Ruby script, which acts as an adapter between the hardware itself and the database. I also used Jupyter notebooks and theJupyter IRuby Kernelto do more in-depth data analysis.✨Editor's Note:Check out this videoto learn how to wrap TimescaleDB functions for the Ruby ecosystem.“Grafana and Timescale really go hand in hand, and both of them are excellent open-source tools with rich documentation”I do use Grafana extensively! Grafana and Timescale really go hand in hand, and both of them are excellent open-source tools with rich documentation. Both have Docker images and can be set up quickly and easily. It is great to plot the measurements with just a few SQL queries.​Without Grafana, I would have had to write a lot of the plots myself, and correlating different time-series events with each other would have been much harder. Either I would have had to spend more time implementing that myself, or I wouldn't have gotten this level of information from my measurements.​The main benefit of using TimescaleDB is that you get a well-balanced mixture. Being built on top of PostgreSQL and still giving you access to all regular relational database features, you can use TimescaleDB for a much larger variety of tasks.​You can also dive right in, even with minimal SQL knowledge—and if you ever do get stuck, theTimescaleandPostgreSQL documentationare well written and extensive, so you can almost always find a solution!​Lastly, there is no ""too small"" for TimescaleDB either. Being open source, quick to set up, and easy to use while performing well even on a spare laptop or PC, it can be a valuable tool for any sort of data acquisition—even if it feels like a ""small"" task!​I learned a lot about my system already by using TimescaleDB and can be much more confident in how to proceed thanks to it. In a way, it has even changed my developer experience. I have the data acquisition and TimescaleDB running continuously while working on new software features or modifying the hardware. If I were to introduce a bug that could mess up the measurements, I might see those much sooner in the data. I can then react appropriately and quickly while still developing.​I can worry less about ensuring everything works and focus more on adding new features!Advice and ResourcesGive TimescaleDB a try whenever you need to record any kind of time-series data. It's much easier to use than writing your own script or dumping it into a CSV file. The built-in functionality can take care of a lot of number crunching very efficiently, allowing you to benefit from years of finely developed functions. It's well worth the trouble, and there's no such thing as ""too small"" a use case for it!​TimescaleDB has helped me be more confident in developing my hardware, and I want to motivate others in my field to try it out themselves. A lot can be gained from a few days or months of data, even if at first you don't think so—some insights only pop out at larger scales, no matter what you're working on.I am very excited to share my use case of TimescaleDB. Whenever I think of databases, I am drawn to these larger web services and big tech companies—but that doesn't have to be the case! That can also be helpful advice for others looking for the right database for their use case.We’d like to thank David and all of the folks at the Max Planck Institute for Plasma Physics for sharing their story on how they’re monitoring the Wendelstein 7-X fusion reactor’s heating system using TimescaleDB.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/using-iot-sensors-timescaledb-and-grafana-to-control-the-temperature-of-the-nuclear-fusion-experiment-in-the-max-planck-institute/
2023-02-28T15:44:16.000Z,How Ndustrial Is Providing Fast Real-Time Queries and Safely Storing Client Data With 97 % Compression,"Ndustrialis based out of Raleigh, North Carolina, in the United States. Our company has been around since 2011, working exclusively in the industrial sector.We aim to help our customers—who are large industrial energy users—save energy. And by doing so also save money and improve sustainability. A lot of that is collecting data from very disparate devices, such as industrial IoT equipment, utility meters, utility bills, and even external data, and ultimately trying to use and combine that data to give our customers a better view of what we would call production-normalized KPIs.The company’s high-level data architectureThese KPIs are essentially metrics that we try to get down to the point of the single unit that our customers are producing, and that changes based on their manufacturing process, what sector they're in, and what their actual product is. But if we can get it down to that point, we can understand how much energy consumption is required for a single product or unit. And then help them find and even completely automate these opportunities to save energy.We’re talking about customers that might have multiple facilities, even in the hundreds across the US and the world, so our goal in this space is to be able to aggregate all that data into a single view, understand what’s happening, and where you may have a lack of efficiencies in order to find those opportunities and save energy and money.The Ndustrial team building more than data applicationsWe currently have around 18 engineers working at the company, split into two main teams. We have an Integrations team that's more focused on different types of ingestion and where we get that data from (which sometimes gets very specific with our customers). The other team is focused on building our platform's more generalized extensibility aspect and a front end,our energy intensity user interface (UI), called Nsight.I am the technical lead and software architect for the Platform team. We focus on building the functionality to turn the customers’ data into meaningful metrics based on their organization, as well as building out features for Nsight, making that easier to consume as a customer.Implementing Real-Time AggregationsMore than 80 percent of our data is time series across the board. Now, the facets of where it comes from could be standard IoT devices just setting up their data points. It could be a time series of line items on a utility bill associated with a statement per day, per month, or per year. But it always comes down to data points associated with a timestamp that we can aggregate together to give a perspective of what’s happening.✨Editor’s Note:Learn more about the features of the best database for time-series data.Then, we have the other side: non-time series, to give a digital perspective of the client’s organization so you can start associating different elements by those groupings or categories to provide even more focused metrics in our UI.The company’s UI, NsightI joined Ndustrial in June 2020, and we already had some applications to store and manage time-series data. This data is very unique, frequent, and needs to be retrieved quickly. But as far as leaning on databases specific to time-series data, that wasn’t really a factor in the beginning. It was more focused on, “How can we store this and at least get to it quickly?”“Compression was a game-changer from our perspective: not having to worry about getting databases on the order of 5, 10, or 15 TB to store this information was a massive factor for us”But then you start considering other aspects: now, I want to aggregate that data in unique ways or across arbitrary windows. That's where the drive to “We need something more powerful, something that can handle this data and not just store it efficiently but query it efficiently” comes in. And along with it came Timescale.Compression was a game-changerfrom our perspective as a company: not having to worry about getting databases on the order of 5, 10, or 15 TB to store this information was a massive factor for us. But then we want the dashboarding that we have, Nsight, which is our energy intensity platform, to be very dynamic.We don’t always know ahead of time what our customers want to visualize. We’re not trying to hard code a bunch of metrics we can report on quickly. We tried to build a much more dynamic platform, and most of our querying is actually on demand. There is some downsampling in between to make things more efficient, but the queries we are providing in our dashboard and arbitrary metrics are all things our customers are creating, and they happen on demand.Getting data back efficiently without feeling like the UI is slow was a major win for us in using TimescaleDB. Before, we had a lot more pre-aggregation. That was the biggest power I've seen from it, along with not having to deal with having an entire team of database administrators (DBAs) to go through and make sure that's possible [Ndustrial uses Managed Service for TimescaleDB]. It’s also one of the more valuable benefits for our customers, whether or not they realize this is even happening.Achieving 97% Compression of Postgres Data Using TimescaleDBWhen I joined the company, Ndustrial had a proof-of-concept (PoC) running for TimescaleDB, so I think the company as a whole was moving in that direction. I had some experience with TimescaleDB and other time-series databases. This was only three years ago, but even then, Timescale was very prominent in the world of “Hey, we need a time-series database to be stored somewhere.”“For one of our larger customers, we normally store about 64 GB of uncompressed data per day. With compression, we’ve seen, on average, a 97 percent reduction”Other alternatives we researched were InfluxDB and Prometheus for storing this data. But what brought it home for us is that PostgreSQL is a foundation in our company. It is a standard of what we expect from developers and, ultimately, the main reason the PoC started to test out how we could work with time-series data without changing and bringing up new technologies and languages to learn.We knew we would be collecting many times-series points per day—across all the different types of data we’re collecting, above a hundred million points per day. It gets massive, and we want to store and query that efficiently. It’s not just about how we store this and how we ensure we’re not spending mass quantities of money to store it, but also how we get that back out, ask questions, and query data without having to do a lot of work in between to get it into a shape where we can do that efficiently.So, the compression aspect of TimescaleDB was something we leaned on immediately. Honestly, that was probably what sold it out of the PoC: when we tested compression with our chunks and started seeing the percentage by which we could reduce the storage size with minor degradation in query performance.✨Editor’s Note:Read how TimescaleDB expands the PostgreSQL functionality with faster queries and reduces storage utilization by 90 percent.For one of our larger customers, which would take a big chunk of the time series, we normally store about 64 GB of uncompressed data per day. With compression, we’ve seen, on average, a 97 percent reduction. So down to less than 2 GB a day. And that goes from a record count of two million records to just a couple hundred thousand, which is pretty significant. It would take a larger and more performant database to hold that type of weight.CREATE FUNCTION public.show_all_chunks_detailed_size()
    RETURNS TABLE(
        hypertable text,
        chunk text,
        time_range tstzrange,
        total_bytes bigint,
        total_size text,
        table_size text,
        index_size text,
        toast_size text,
        compression_savings numeric
     ) AS
$func$
BEGIN
    RETURN QUERY EXECUTE (
      SELECT string_agg(format('
            SELECT
                %L AS hypertable,
                s.chunk_schema || ''.'' || s.chunk_name AS chunk,
                tstzrange(c.range_start, c.range_end) AS time_range,
                s.total_bytes,
                pg_size_pretty(s.total_bytes) AS total_size,
                  pg_size_pretty(s.table_bytes) AS table_size,
                pg_size_pretty(s.index_bytes) AS index_size,
                pg_size_pretty(s.toast_bytes) AS toast_size,
                round(100 * (1 - p.after_compression_total_bytes::numeric / p.before_compression_total_bytes::numeric), 2) AS compression_savings
            FROM
                chunks_detailed_size(%L) s
                LEFT JOIN chunk_compression_stats(%L) p USING (chunk_name)
                LEFT JOIN timescaledb_information.chunks c USING (chunk_name)
            ', tbl, tbl, tbl
        ), ' UNION ALL ')
        FROM (
            SELECT hypertable_schema || '.' || hypertable_name AS tbl
            FROM timescaledb_information.hypertables
            ORDER BY 1
        ) sub
   );
END
$func$ LANGUAGE plpgsql;The query that the Ndustrial team usesto monitor their chunk sizes across hypertables, including the compression stats they are seeingMost companies believe this, but data is critical to us. We keep raw data: we never know how we will go back and reuse it. It's not something we can downsample, throw away, and hope one day we don't need it. So being able to store it and not having to focus too much on complex retention strategies to put things in a much slower, cold storage (versus hot storage) because we're able to compress it and still query it without worrying too much about going over limits is very significant for us.Replacing Cassandra for time-series dataAs far as query performance, I don't have an exact metric. Still, being able to make arbitrary queries, arbitrary roll-ups, on-demand unit conversion, and currency conversion at the time of querying and get responses back in the same amount of time that we got from doing pre-aggregates and without the difficulty of changing the questions’ format to roll up this data is a massive benefit.Before TimescaleDB, we were using Cassandra, and a fair amount of mixed data could also have been in a standard PostgreSQL database. But most of our high-frequency IoT data was in Cassandra, primarily for the benefit of being able to use partitioning for fast retrieval. However, it took a lot more effort to get that data out. So as efficient as it could be to store it, we had difficulty on the other side of getting it back out in the way that we wanted it.We also tried InfluxDB but didn’t go with it. It wasn’t just about learning a new language (InfluxQL); storing most of our relational data inside PostgreSQL was a significant factor. The platform we've been working with for the last two years has been a shift in usingGraphQLas our main layer for that service, our tenancy.TimescaleDB provides the time-series metrics, but the rest of our relational graph that builds out an organization are just entities and our PostgreSQL database. So, being able to use time-series metrics as they relate to relational entities, groupings, and various ways of querying those types of data without going outside PostgreSQL, was a significant aspect for us.With InfluxDB, we would deal with more latency: the amount of time it would take to query on one side, query on the other side, and correlate them together. Not to mention that you’re dealing with a new language. With TimescaleDB, it’s just a SQL query.I’ve had experience with InfluxDB, which is one of the reasons we made sure we tested that out as well. And that's probably a direction I would go back to if TimescaleDB didn’t exist. Luckily, we don’t have to. I mean, it's also a powerful database, but for the other factors I just mentioned, TimescaleDB made more sense at the time.Continuous Aggregation andtime_bucket_gapfillFrom a more architectural perspective, I'm focused on ensuring the technologies we bring in aren't going to hinder our developers by having to make them learn something new every time we bring in a new engineer. If there's too much of a learning curve and it's not something they're already used to, you're going to slow down.The Ndustrial data architectureAs a small startup, we must balance the technologies we can benefit from without having 10 developers purely stationed for managing them, which is why we chose Managed Service for TimescaleDB. You don't want to have every developer spend all their time managing it or being aware of it and worrying about what happens if you don’t apply those maintenance updates or backups correctly.“Getting all these updates and zero downtime for deployments and replicas is significant for us because we need to assure our customers that we are as reliable as the data we're storing”Time series is the main portion of the type of data we collect, so it’s a very significant aspect of our platform. We wouldn’t be able to do much without it. Putting that in the hands of the same developers that are also having to focus on integrations, customers, and future work would have been too much of a risk. On the other hand, getting all these updates and zero downtime for deployments and replicas is significant for us because we need to assure our customers that we are as reliable as the data we're storing.As for deployment, on the ingestion side, we useKotlinas our foundation for building out stream-processing pipelines on a technology calledPulsaras our messaging layer. It's very similar toKafkaPTP-style (point-to-point) message queuing. Essentially, all of the ingestion that we do, whether it's IoT data or other types of integrations that we're either pulling from or getting pushed to, flows through these stream processing pipelines, ultimately landing into our TimescaleDB database. So that is our main input into TimescaleDB.Then, on the other side of the house, we have our GraphQL platform. The platform exposes a GraphQL API that can query TimescaleDB for arbitrary queries, metrics, and results based on what the consumer is asking. And then, in our front end, which is our dashboarding, there’s a full-featured single-page app in React.We also use other technologies, such asGrafana, for a more ad hoc view of the data, especially because we also do implementations at facilities for installing meters and devices if the customers don't already have those available. Hooking those up and having a view into our TimescaleDB database of that specific feed of data gives us a lot more visibility and speeds up the installation time without worrying about whether or not it's set up correctly.Our data is very disparate and unpredictable in terms of how it gets sent. Some customers will have devices sending data every second to minutes or every hour. Devices might go down, come back up, and send their last days’ worth of data all at once. We could have other systems sending it at their own regular rates.It's not normalized, so to be able to lean on TimescaleDB hyperfunctions liketime_bucket_ gapfilland bucketing, andlocftype of functions to turn those results into something more meaningful because we can fill the gap or understand what should happen in the case where that data point didn't exist is really powerful for us.✨Editor’s Note:Learn more about hyperfunctions and how they can help you save time and analyze time-series data better.Obviously, compression was a big thing for us in terms of getting that in place in the beginning. And since then, we have used continuous aggregates. We are still at a point where we're building out the process of what that will look like in our platform because this data is unique—I mean, we're storing arbitrary data. We don't even know sometimes what that feed is sending us.We’re still building a more automated process to be able to make smarter decisions about what becomes continuous aggregates, what becomes something we physically want to store, and the intervals at which we want to store it and throw away the rest. And so, I hope to lean on continuous aggregates a bit more in the future.But, ultimately, just being able to query and use TimescaleDB features to roll up and aggregate that data is significant for us. I love following theTimescale experimental repositoryand seeing what people come out with each release. I basically have a list on my computer of things I intend to add to our platform soon. Just to expose those capabilities, too. Because if we can do it, why not turn that into something we can create in our app? And create it as another view, metric, or perspective in the UI. So it's always exciting to see what you guys are coming up with.Creating an In-House Metrics EngineOver the last few years, we've been building this new platform and leaning pretty heavily on it. There are a few pieces left that we're getting to really make it where we want it to be. Part of that is exposing many of these self-made dynamic metrics to our customers to the degree where they can just generate whatever they want that meets the needs of their system based on their use case or personas.We have various individuals using our platform that come from different disciplines within an organization, so being able to cater metrics and analytics to them directly is really powerful, and exposing a platform in a way where they can do that without having to rely on us to do it for them is also very powerful.To achieve this, we're building more of these ad hoc type views, which will lean on the functionality we get from an in-house metrics engine that uses TimescaleDB to query this data. Beyond that, we're entering into more of an automated controls world within our platform.Up to this point, we've primarily been focused on getting this data visible and exposing it to our customers, showing them how they can become more efficient, and working with them more closely to help them get there. But we're moving toward the direction of taking these opportunities and automating the process that allows them to save energy and money or participate in these opportunities.This requires us to analyze this data to predict outcomes that could happen in the near future so that we can prepare customers and get them set up and make them more valuable than others who don't have that information. It’s not really a competition, but at the same time, we want to show clients that they can get value out of their data. So having all of our time-series data in a common place that we can query in a common way is going to really make that successful in the future.Advice & ResourcesMy advice is always to start with the type of data you’re storing. In the beginning, you obviously don't want to break things out too early. You never know what could shift. Wait for that next use case before you start extracting or breaking things out: you never know if you’ll end up doing it, and then you might just have more complexity on your hands for no reason.But as a whole, data is different across the board. It's okay to store it in different places that are meant to store that data. So, having a time-series database versus storing a relational database over here and maybe search indexes over here. That's okay. I mean, is your product going to be piecing those together and really leaning on the type of data I'm storing? What is the size and frequency of this data?And then selecting the right technologies specific to that data or maybe multiple technologies to handle that data. You should do that ahead of time because it gets more difficult the longer you wait to do so because it becomes the process of migrating all of that and doing it without downtime. But ultimately, that's just the best way to start: understand what you're storing and figure it out from there.Regarding resources, I think that theTimescale Docsare great. Especially dealing with other technologies and being a developer, there are many times you’ll go to the documentation for some technology, and you’ll be as confused as you were before you got there. I think Timescale has done a wonderful job with their docs and how they break it out into use cases and some examples.I also lean pretty heavily onCommunity Slack, whether or not I'm asking a question. It's just nice to monitor it and see what other people are asking. It might raise questions or give me ideas about issues we ran into in the past. And so having that as a resource as well, or knowing that if I have an issue, I can ask and get pretty quick and immediate feedback. It's really great.Between the Docs and the Community, it shouldn’t be difficult to get up with what you need to meet your system’s needs.Want more insider tips?Sign up for our newsletterfor more Developer Q&As, technical articles, and tutorials to help you do more with your data. We deliver it straight to your inbox twice a month.We’d like to thank Michael and the folks at Ndustrial for sharing their story on how they speeded up their client-facing dashboards, even for real-time queries, while carefully storing all their customers’ data by making the most of TimescaleDB’s compression powers.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-ndustrial-is-providing-fast-real-time-queries-and-safely-storing-client-data-with-97-compression/
2023-02-14T14:30:00.000Z,From Ingest to Insights in Milliseconds: Everactive's Tech Transformation With Timescale,"📬This blog post was originally published in February 2021 and updated in February 2023 to reflect Everactive's data stack and business evolution.This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Carlos Olmos,Dan Wright, andClayton Yochumfrom Everactive join us to share how they’re bringing analytics and real-time device monitoring to scenarios and places never before possible. Learn how they’ve set up their data stack (moving from six to only three instances), their database evaluation criteria, their advice for fellow developers, and more.About the CompanyEveractivecombines battery-free, self-powered sensors and powerful cloud analytics to provide “end-to-end” hyperscale IoT solutions to our customers. Our company is undergoing a huge transformation from focusing only on industrial monitoring services (read more about Industrial IoT) to opening our platform to developers that can leverage our technology to create their own solutions and services.It is not easy to build a platform, less to build an open platform. Flexibility and stability are key factors. We have found so far that with Timescale (and PostgreSQL underneath), we have been able to bend our data models and data serving needs to implement the new use cases for the platform without pain.We design and build our sensors in-house (down to the chip), and they’re ruggedized for harsh settings and can operate indefinitely from low levels of energy harvested from heat or light. This means our customers’ devices can continuously stream asset health data, despite radio interference and physical obstacles—like equipment, ducts, and pipes—common in industrial settings.Since they charge themselves, these sensors stay operational well beyond what’s possible with traditional, battery-powered Industrial IoT devices. We ingest data from thousands of sensorsinto Timescale, then surface it to our customers through dashboards, charts, and automated alerts.Ourinitial productsare designed to monitor steam systems, which are used in various industries and applications, like process manufacturing, chemical processing, and district energy, as well as a range of rotating equipment, such as motors, pumps, fans, and compressors. Currently, we serve large, Fortune 500 manufacturers in many sectors, including Food & Beverage, Consumer Packaged Goods, Chemical Process Industries, Pharmaceuticals, Pulp & Paper, and Facilities Management.We show customers their data through a web-based dashboard, and we also have internal applications to help our in-house domain experts review and label customer data to improve our automated failure detection.About the TeamWe’re a small team of software and data engineers, spanning the Cloud and Data Science teams at Everactive.Between us, we’ve got several decades of experience managing databases, pipelines, APIs, and various other bits of backend infrastructure.About the ProjectOur key differentiator is that our sensors are batteryless: the custom low-power silicon means that they can be put in more places, without requiring servicing for well over a decade.In turn, this means that we can monitor factory devices that were formerly cost-prohibitive to put sensors on, due to the difficulty or cost associated with charging batteries; being able to collect dataeconomicallyfrom more equipment also means that our industrial data streams are more detailed and cover more equipment than our competitors’.Today, customers place our sensors on steam traps and motors, and we capture a range of metrics – from simple ones, like temperature, to more complex ones, like 3D vibrational data. (You can learn more about steam trap systems and the need for batteryless systems inthis overview video.)Everactive’s new generation of sensors in the wild, including a sensor on a hydraulic arm with a shipping container and, below, a sensor on a shipping container clasp closer upWe then use this data to inform our customers about the health of their industrial systems, so they can take action when and where required. “Action” in this sense could mean replacing a steam trap, replacing a bad bearing in a machine, or various other solutions to problems.For example, we’ll automatically alert customers if their monitored equipment has failed or if machines are off when they should be on, so customers can send a crew to fix the failure, or power on the machine remotely.In addition to receiving alerts from us, customers can use our dashboards to check the latest data and current status of their equipment at any time.One of Everactive’s dashboards, tracking the overall vibration levelAs mentioned earlier, our team’s responsible for delivering these intuitive visualizations to our customers and in-house domain experts –  as well as for feeding sensor metrics into our custom analytics to automate failure detection and improve our algorithms.Using (and Choosing!) TimescaleDBBefore TimescaleDB, we stored metadata in PostgreSQL and our sensor data in OpenTSDB. Over time, OpenTSDB became an increasingly slow and brittle system.Our data is very well-suited to traditional relational database models: we collect dozens of metrics in one packet of data, so it makes sense to store those together. Other time-series databases would force us to either bundle metrics into JSON blobs (making it hard to work with in-database) or to store every metric separately (forcing heavy, slow joins for most queries of interest).TimescaleDB was an easy choice because it let us double down on PostgreSQL, which we already loved using for metadata about our packet streams.We looked briefly at competitors like Influx but stopped considering them once it was clear TimescaleDB would exceed our needs.Our evaluation criteria were pretty simple: will it handle our load requirements, and can we understand how to use it? The former was easy to test empirically, and the latter was essentially “free” as TimescaleDB is “just” a PostgreSQL extension.This “just PostgreSQL” concept also lets us carefully manage our schema as code, testing and automating changes through CI/CD pipelines. We usesqitch, but popular alternatives includeFlywayandLiquibase. We like sqitch because it encourages us to write tests for each migration and is lightweight (no JVM).We previously usedAlembic, the migration component of the popularSQLALchemy Python ORM, but as our TimescaleDB database grew to support many clients, it made less sense to tie our schema management to any one of them.We maintain a layer of abstraction within TimescaleDB by separating internal and external schemas.“The capacity of Timescale to support both traditional schemas and time-series data in the same database allowed us to consolidate into one storage solution”Our data is stored as (hyper)tables in internal schemas like “packets” and “metadata,” but we expose them to clients through an “API” schema only containing views, functions, and procedures. This allows us to refactor our data layout while minimizing interruption in downstream systems by maintaining an API contract. This is a well-known pattern in the relational database world—yet another advantage of TimescaleDB being “simply” a PostgreSQL extension.Current Deployment & Future PlansAnother example of an Everactive dashboardWe useTimescaleand love it. We already used PostgreSQL on Amazon RDS and didn’t want to have to manage our own database (OpenTSDB convinced us of that!).It had become normal for OpenTSDB to crashmultiple times per weekfrom users asking for slightly too much data at once.TimescaleDB is clearly much faster than our previous OpenTSDB system. More importantly, nobody has ever crashed it.One not-very-carefully-benchmarked but huge performance increase we’ve seen?We have a frontend view that requires the last data point from all sensors: in OpenTSDB, it required nearly 10minutesto load (due to hard-to-fix tail latencies in HBase), and our first TimescaleDB deployment brought that down to around 7seconds. Further improvements to our schema and access patterns have brought these queries into the sub-second range.""We moved those schemas from Amazon RDS for PostgreSQL to Timescale. The migration was very simple: moving among PostgreSQL schemas was straightforward""We have been able to maintain sub-second responses even with the growth of data volume: that's very good for us. Also, thanks to compression andcontinuous aggregates, we have been keeping our table sizes in check and with great performance.✨Editor’s Note:For more comparisons and benchmarks, seehow Timescale compares to Amazon RDS for PostgreSQL. To learn more and try Timescale yourself, see ourstep-by-step Timescale tutorial.Timescale has been so good for us that it’s triggered a wave of transitions to managed solutions for other parts of our stack.We’ve recently moved our Amazon RDS data into Timescale to further simplify our data infrastructure and make it easier and faster to work with our data.About 20 % of our data is metadata kept in a relational database. We moved those schemas from Amazon RDS for PostgreSQL to Timescale. The migration was very simple: moving among PostgreSQL schemas was straightforward.We chose RDS from the beginning for simplicity. Eventually, once we had Timescale up and running, it became evident that we didn't need two separate PostgreSQL vendors when we were having such good results with Timescale.The capacity of Timescale to support both traditional schemas and time-series data in the same database allowed us to consolidate into one storage solution. The instances multiply because we keep three environments (development, staging, and production) for each database, so we went from six (three RDS plus three Timescale) to only three Timescale instances.As you’ll see in the below diagram, our sensors don’t talk directly to TimescaleDB; they pass packets of measurements to gateways via our proprietary wireless protocol. From there, we useMQTTto send those packets to our cloud.From our cloud data brokers,Kafkaprocesses and routes packets into TimescaleDB (and Timescale), and our TimescaleDB database powers our dashboard and analytics tools. We also added a third component: outbound channels for our platform users.Everactive architecture diagramCompared to Amazon RDS for PostgreSQL, Timescale also consolidates a lot of the costs associated with operating our instance in AWS—we now have a simpler bill that makes it easier to forecast costs.From the operation point of view, dealing with fewer instances and relying more on the Timescale Support Team for infrastructure maintenance has reduced our database maintenance workload significantly. Our security operations also benefited from the migration, again thanks to the consolidation and the transfer of certain responsibilities to Timescale.✨Editor’s Note:Read how our Support Team is raising the baron hosted database support.We’ll continue to innovate on our technology platform and increase Everactive’s product offerings, including improving our sensors’ wireless range, lowering power requirements to increase energy harvesting efficiency, integrating with additional sensors, and shrinking device form factor. These successive chip platform enhancements will allow us to monitor the condition of more and more assets, and we’re also developing a localization feature to identifywhereassets are deployed.Ultimately, Everactive’s mission is to generate new, massive datasets from a wealth of currently un-digitized physical-world assets. Transforming that data into meaningful insights has the potential to fundamentally improve the way that we live our lives—impacting how we manage our workplaces, care for our environment, interact with our communities, and manage our own personal health.Advice & ResourcesIf you’re evaluating your database options, here are two recommendations based on our experiences. First, if you have enough time-series data that a general database won’t cut it (millions of rows), TimescaleDB should be your first choice. It’s easy to try out,the docs are great, and thecommunityis very helpful.Second, don’t underestimate the importance of using solutions that leverage a wide knowledge base shared by many/most backend developers. The increase in team throughput and decrease in onboarding time afforded by TimescaleDB—everyone knows at leastsomeSQL—in contrast to OpenTSDB—an esoteric thing built on HBase—has been a huge advantage. We expected this to some degree, but actually experiencing it firsthand has confirmed its value.Additionally, the use of schema-as-code tools and an internal/external schema separation discussed above have also been cornerstones of our success. We hadn’t been using these tools and patterns at Everactive previously but have since seen them catch on in other projects and teams.Want to read more developer success stories?Sign up for our newsletterfor more Developer Q&As, technical articles, tips, and tutorials to do more with your data—delivered straight to your inbox twice a month.We’d like to thank Carlos, Dan, Clayton, and all of the folks at Team Everactive for sharing their story (special shoutout to Brian, Joe, Carlos, Greg, and Elise for your contributions to this piece!). We applaud your efforts to bring sustainable, cost-effective sensors and easily actionable data to organizations around the world 🙌.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-everactive-powers-a-dense-sensor-network-with-virtually-no-power-at-all/
2022-12-28T14:30:05.000Z,How Newtrax Is Using TimescaleDB and Hypertables to Save Lives in Mines While Optimizing Profitability,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Jean-François Lambert, lead data engineer atNewtrax, shares how using TimescaleDB has helped the IoT leader for underground mines to optimize their clients’ profitability and save lives by using time-series data in hypertables to prevent human and machine collisions.About the CompanyNewtrax believes the future of mining is underground, not only because metals and minerals close to the surface are increasingly rare but because underground mines have a significantly lower environmental footprint. To accelerate the transition to a future where 100 percent of mining is underground, we eliminate the current digital divide between surface and underground mines.To achieve this goal, Newtrax integrates the latest IoT and analytics technologies to monitor and provide real-time insights on underground operations, including people, machines, and the environment. Newtrax customers are the largest producers of metals in the world, and their underground mines rely on our systems to save lives, reduce maintenance costs, and increase productivity. Even an increase of 5 percent in overall equipment effectiveness can translate to millions in profits.We collect data directly from the working face by filling the gaps in existing communications infrastructures with a simple and easy-to-deploy network extension. With that, we enable underground hard rock mines to measure key performance indicators they could not measure in real time to enable short-interval control of operations during the shift.Our headquarters are based in Montreal, Canada, and we have regional offices around the globe, totaling 150 employees, including some of the most experienced engineers and product managers specializing in underground hard rock mining. Our solutions have been deployed to over 100 mine sites around the world.About the TeamThe Newtrax Optimine Mining Data Platform (MDP) is the first AI-powered data aggregation platform enabling the underground hard rock mining industry to connect all IoT devices into a single data repository. Our team consists of software developers, data scientists, application specialists, and mining process experts, ensuring our customers get the exact solutions they require.As the lead data engineer, it is my responsibility to define, enhance and stabilize our data pipeline, from mobile equipment telemetry and worker positioning to PowerBI-driven data warehousing via RabbitMQ, Kafka, Hasura, and of course, TimescaleDB.About the ProjectA diagram of the Newtrax solutions and how they bring together people, machines, and the environment in their mine intelligence operationsOur Mining Data Platform (MDP) platform incorporates proprietary and third-party software and hardware solutions that focus on acquiring, persisting, and analyzing production, safety, and telemetry data at more than one hundred underground mine sites across the world.From the onset, we have had to deal with varying degrees of data bursts at completely random intervals. Our original equipment manufacturer-agnostic hardware and software solutions can acquire mobile equipment telemetry, worker positioning, and environmental monitoring across various open platforms and proprietary backbones. WiFi, Bluetooth, radio-frequency identification (RFID), long-term evolution (LTE), leaky feeder, Controller Area Network (CAN) bus, Modbus—if we can decode it, we can integrate it.“We have saved lives using TimescaleDB”Regardless of skill sets, programming languages, and transportation mechanisms, all of this incoming data eventually makes it to PostgreSQL, which we have been using since version 8.4 (eventually 15, whenever TimescaleDB supports it!). For the past 10 years, we have accumulated considerable experience with this world-class open-source relational database management system.✨Editor’s Note:Stay tuned as we’ll announce support for PostgreSQL 15 very soon, in early 2023.While we are incredibly familiar with the database engine itself and its rich extension ecosystem, partitioning data-heavy tables was never really an option because native support left to be desired, and third-party solutions didn’t meet all of our needs.Choosing (and Using!) TimescaleDBI have been using PostgreSQL since I can remember. I played around with Oracle and MSSQL, but PostgreSQL has always been my go-to database. I joined Newtrax over 12 years ago, and we’ve used it ever since.In 2019, we found out about TimescaleDB (1.2.0 at the time) and started closely following and evaluating the extension, which promised to alleviate many of our growing pains, such as partitioning and data retention policies.Not only did TimescaleDB resolve some of our long-standing issues, but itsdocumentation,blog posts, and community outlets (Slack,GitHub,Stack Overflow) also helped us make sense of this “time-series database” world we were unknowingly part of.“One of our first tech debt reductions came via the use of thetime_bucket_gapfillandlocffunctions, which allowed us to dynamically interpolate mobile equipment telemetry data points that could be temporarily or permanently missing”A tipping point for choosing TimescaleDB was theNYC TLC tutorialwhich we used as a baseline. After comparing with and without TimescaleDB, it became clear that we would stand to gain much from a switch: not only from the partitioning/hypertable perspective but also with regards to the added business intelligence API functionality, background jobs,continuous aggregates, and variouscompressionandretention policies.Beyond that, we only ran basic before/after query analysis, because we know that a time-based query will perform better on a hypertable than a regular table, just by virtue of scanning fewer chunks. Continuous aggregates and hyperfunctions easily saved us months of development time which is equally, if not more important, than performance gains.More importantly, in order for our solutions (such as predictive maintenance and collision avoidance) to provide contextualized and accurate results, we must gather and process hundreds of millions of data points per machine or worker, per week or month, depending on various circumstances. We usehypertablesto handle these large datasets and act upon them. We have saved lives using TimescaleDB.Beyond the native benefits ofusing hypertablesand data compression/retention policies, one of our first tech debt reductions came via the use of thetime_bucket_gapfillandlocffunctions (see an example later), which allowed us to dynamically interpolate mobile equipment telemetry data points that could be temporarily or permanently missing. Since then, we have been acutely following any and all changes to the hyperfunction API.✨Editor’s Note:Read how you can write better queries for time-series analysis using just SQL and hyperfunctions.Current Deployment & Future PlansAs far as deployment is concerned, the mining industry is incredibly conservative and frequently happens to be located in the most remote areas of the world, so we mostly deploy TimescaleDB on-premises throughKubernetes. The main languages we use to communicate with TimescaleDB are C#, Golang, and Python, but we also have an incredible amount of business logic written as pure SQL in triggers, background jobs, and procedures.While all of our microservices are typicallyrestricted to their own data domains, we have enabled cross-database queries and mutations throughHasura. It sits nicely on top of TimescaleDB and allows us to expose our multiple data sources as a unified API, complete with remote relationships, REST/GraphQL API support, authentication, and permission control.Our use of TimescaleDB consists of the following:We greatly appreciatetimescaledb-tuneautomatically tweaking the configuration file based on the amount of memory and number of CPUs available. In fact, Newtrax has contributed to this tool a few times.We deploy thetimescaledb-haimage (which we also contributed to) because it adds multiple extensions other than TimescaleDB (for example,pg_stat_statements,hypopg,  andpostgis), and it comes with useful command-line tools likepgtop.Our developers are then free to use hypertables or plain tables. Still, when it becomes obvious that we’ll want to configure data retention or create continuous aggregates over a given data set,hypertables are a no-brainer.Even if hypertables are not present in a database, we may still use background jobs to automate various operations instead of installing another extension likepg_cron.Hypertables can be exposed as-is through Hasura, but we may also want to provide alternative viewpoints through continuous aggregates or custom views and functions.✨Editor’s Note:The TimescaleDB tuning tool helps make configuring TimescaleDB a bit easier. Read our documentation to learn how.As mentioned earlier, incoming data can be erratic and incomplete. We might be missing some data points or values. Using a combination of the following code will create a functionmine_data_gapfill, which can be tracked with Hasura and enable consumers to retrieve consistent data series based on their own needs. And you could easily useinterpolateinstead oflocfto provide interpolated values instead of the last one received.CREATE TABLE mine_data
(
  serial TEXT,
  timestamp TIMESTAMPTZ,
  values JSONB NOT NULL DEFAULT '{}',
  PRIMARY KEY (serial, timestamp)
);
SELECT CREATE_HYPERTABLE('mine_data', 'timestamp');

INSERT INTO mine_data (serial, timestamp, values) VALUES
('123', '2020-01-01', '{""a"": 1, ""b"": 1, ""c"": 1}'),
('123', '2020-01-02', '{        ""b"": 2, ""c"": 2}'),
('123', '2020-01-03', '{""a"": 3,         ""c"": 3}'),
('123', '2020-01-04', '{""a"": 4, ""b"": 4        }'),
('123', '2020-01-06', '{""a"": 6, ""b"": 6, ""c"": 6}');


CREATE FUNCTION mine_data_gapfill(serial TEXT, start_date TIMESTAMPTZ, end_date TIMESTAMPTZ, time_bucket INTERVAL = '1 DAY', locf_prev INTERVAL = '1 DAY')
RETURNS SETOF mine_data AS $$
  SELECT $1, ts, JSONB_OBJECT_AGG(key_name, gapfilled)
  FROM (
    SELECT
      serial,
      TIME_BUCKET_GAPFILL(time_bucket, MT.timestamp) AS ts,
      jsondata.key AS key_name,
      LOCF(AVG((jsondata.value)::REAL)::REAL, treat_null_as_missing:=TRUE, prev:=(
        SELECT (values->>jsondata.key)::REAL
         FROM mine_data
        WHERE values->>jsondata.key IS NOT NULL AND serial = $1
        AND timestamp < start_date AND timestamp >= start_date - locf_prev
        ORDER BY timestamp DESC LIMIT 1
      )) AS gapfilled
    FROM mine_data MT, JSONB_EACH(MT.values) AS jsondata
    WHERE MT.serial = $1 AND MT.timestamp >= start_date AND MT.timestamp <= end_date
    GROUP BY ts, jsondata.key, serial
    ORDER BY ts ASC, jsondata.key ASC
  ) sourcedata
  GROUP BY ts, serial
  ORDER BY ts ASC;
$$ LANGUAGE SQL STABLE;

SELECT * FROM mine_data_gapfill('123', '2020-01-01', '2020-01-06', '1 DAY');You can find this code snippet on GitHub too.RoadmapWe need to start looking into thetimescaledb-toolkitextension, which is bundled in thetimescaledb-haDocker image. It promises to “ease all things analytics when using TimescaleDB, with a particular focus on developer ergonomics and performance,” which is music to our ears.“Honestly, theTimescaleDocshad most of the resources we needed to make a decision between switching to TimescaleDB or continuing to use plain PostgreSQL (this article in particular)”Another thing on our backlog is to investigate using Hasura’s streaming subscriptions as a Kafka alternative for specific workloads, such as new data being added to a hypertable.Advice & ResourcesHonestly, theTimescaleDocshad most of the resources we needed to make a decision between switching to TimescaleDB or continuing to use plain PostgreSQL (this article in particular). But don’t get too swayed or caught up in all the articles claiming outstanding performance improvements: run your own tests and draw your own conclusions.We started by converting some of our existing fact tables and got improvements ranging from modest to outstanding. It all depends on your use cases, implementations, and expectations. Some of your existing structures may not be compatible right out of the box, and not everything needs to become a hypertable either! Make sure to consider TimescaleDB’s rich API ecosystem in your decision matrix.Sign up for our newsletterfor more Developer Q&As, technical articles, tips, and tutorials to do more with your data—delivered straight to your inbox twice a month.We’d like to thank Jean-François and all of the folks at Newtrax for sharing their story on how they’re bringing digital transformation to a conservative industry—all while saving lives along the way, thanks to TimescaleDB and hypertables.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-newtrax-is-using-timescaledb-and-hypertables-to-save-lives/
2021-02-10T18:09:58.000Z,How Conserv Safeguards History: Building an Environmental Monitoring and Preventive Conservation IoT Platform,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Nathan McMinn, CTO and co-founder at Conserv, joins us to share how they’re helping collections care professionals around the world understand their collections’ environments, make decisions to optimize conditions, and better protect historical artifacts - no IT department required.Think for a moment about your favorite museum. All of the objects it houses exist to tell the stories of the people who created them - and of whose lives they were a part - and to help us understand more about the past, how systems work, and more. Without proper care, which starts with the correct environment, those artifacts are doomed to deteriorate and will eventually be lost forever (read more about the factors that cause deterioration).Conservstarted in early 2019 with a mission to bring better preventive care to the world’s collections. We serve anyone charged with the long-term preservation of physical objects, from paintings and sculptures to books, architecture, and more. (You’ll often hear our market segment described as “GLAM”: galleries, libraries, archives, and museums.) At the core, all collections curators need to understand the environment in which their collections live, so they can develop plans to care for those objects in the short, mid, and long-term.While damagecancome in the form of unforeseen and catastrophic events, like fire, theft, vandalism, or flooding, there aremanyissues that proactive monitoring and planning can prevent, such as: mold growth, color fading associated with light exposure, and mechanical damage caused by fluctuating temperature and relative humidity.At Consev, we’ve built a platform to help collections gather the data required to understand and predict long-term risks, get insights into their data, and plan for how to mitigate and improve the preservation environment. Today, we’re the only company with an end-to-end solution - sensors to screen - focused on preventive conservation.  Collections like theAlabama Department of Archives and Historyand various others rely on us to develop a deep understanding of their environments.About the teamWe’re a small team where every member makes a big impact. Currently, we’re at 6 people, and each person plays a key role in our business:Austin Senseman, our co-founder and CEO, often describes his background as “helping people make good decisions with data.” He has extensive experience in analytics and decision support – and a history of using data to guide organizations to their desired outcomes.Nathan McMinn(this is me!), our other co-founder and CTO, comes from the development world. I’ve spent my career leading engineering teams and building products people love, most notably in the enterprise content management sector.Ellen Orr, a museum preparator turned fantastic collections consultant.Cheyenne Mangum, our talented frontend developer.Melissa King, a preventative conservator who recently joined to help us build better relationships with more collections.Bhuwan Bashel, who is joining in a senior engineering role (oh yeah, he’ll get some TimescaleDB experience quickly!).About the projectWe collect several types of data, but the overwhelming majority of the data we collect and store in TimescaleDB is IoT sensor readings and related metrics. Our solution consists of fleets of LoRaWAN sensors taking detailed environmental readings on a schedule, as well as event-driven data (seethis articlefor an overview of LoRaWAN sensors, use cases, and other details).So, what ends up in our database is a mix of things like environmental metrics (e.g., temperature, relative humidity, illuminance, and UV exposure), sensor health data (e.g., battery statistics), and data from events (e.g., leak detection or vibration triggers).We also capture information about our customers’ locations - such as which rooms are being monitored -  and data from human observations - such as building or artifact damage and pest sightings - to give our sensor data more context and some additional “texture.” It’s one thing to collect data from sensors, but when you pair that with human observations, a lot of interesting things can happen.Our UI makes it simple to add a human observation to any data point, collection, or date.See our docsfor more details.For us, it is all about scale and performance.We collect tens of thousands of data points each day, and our users think about their metrics and their trends over years, not days.Also, like anybody else, our users want things to feel fast. So far, we’re getting both from TimescaleDB.With those criteria met, our next challenge is how to use that data to (1) provide actionable insights to our customers, allowing them to ask and answer questions like “what percentage of time is my collection within our defined environmental ranges?” and (2) offer in-depth analysis and predictions, like possible mold and mechanical damage risks.This iswhere we’ve gotten the most value out of TimescaleDB: the combination of a performant, scalable repository for our time-series data, the core SQL features we know and trust, and the built-in time-series functionalityprovided by TimescaleDB let us build new analysis features much faster.Example of our Collections focused analytics dashboard -see our docsfor more details.✨Editor's Note:To learn how we’ve architected TimescaleDB to support joining time-series data with other relational data,check out our data model documentationandfollow our Hello, Timescale! tutorialto try it out for yourself.Using (and choosing) TimescaleDBI first found out about TimescaleDB at anAll Things Open conferencea few years ago; it was good timing, since we were just starting to build out our platform. Our first PoC used ElasticSearch to store readings, which we knew wouldn’t be a permanent solution. We also looked at InfluxDB, and, while Amazon’s Timestream looked promising, it wasn’t released yet.Our database criteria was straightforward; we wanted scale, performance, and the ability to tap into the SQL ecosystem. After evaluating TimescaleDB’s design, we were confident that it would meet our needs in the first two categories, but so would many other technologies.What ultimately won me over was the fact that it’s PostgreSQL.I’ve used it for years in projects of all sizes; it works with everything, and it’s a proven, trustworthy technology – one less thing for me to worry about.✨Editor’s Note:To see how TimescaleDB stacks up to alternatives, read ourInfluxDB vs. TimescaleDBandAmazon Timestream vs. TimescaleDBbenchmark posts (included in-depth performance, scale, and developer experience comparisons) andexplore our at-a-glance comparisons.Current deployment & use casesOur stack is fairly simple, standard stuff for anyone that has a UI → API → database pattern in their application. Our secret sauce is in how well we understand our users and their problems, not our architecture :).Some of our future plans will require us to get more complex, but for now we’re keeping it as simple and reliable as possible:Node.js services running in Docker containers on AWS ECS, with TimescaleDB on the database tierReact.js frontendMobile app built with FlutterIn the near future, I’d like to move over toTimescaleDB’s hosted cloud offering. As we get bigger, that will be something we evaluate.In line with the above, our queries themselves aren’t that clever, nor do they need to be. We make heavy use of PostgreSQL window functions, and the biggest features we benefit from, in terms of developer ergonomics, are TimescaleDB’s built-in time-series capabilities:time_bucket,time_bucket_gapfill,histograms,last observation carried forward, and a handful of others. Almost every API call to get sensor data uses one of those.We haven’t usedcontinuous aggregatesorcompressionyet, but they’re on my to-do list!✨Editor’s Note:In addition to the documentation links above,check out our continuous aggregates tutorialfor step-by-step instructions and best practices, andread our compression engineering blogto learn more about compression, get benchmarks, and more.Getting started advice & resourcesTimescaleDB gives you more time to focus on end-user value, and less time focusing on things like “Can I connect tool x to my store?” or “How am I going to scale this thing?”If you’re considering TimescaleDB or databases in general, and are comfortable with Postgres already, try it. If you want the biggest ecosystem of tools to use with your time-series data, try it.  If you think SQL databases are antiquated and don’t work for these sorts of use cases, try it anyway - you might be surprised.For anybody out there thinking about an IoT project or company, as a technologist, it’s really tempting to focus on everything before data gets to the screen. That’s great, and you have to get those things right, but that’s just table stakes.Anybody can collect data points and put a line graph on a screen. That’s a solved problem.Yourchallenge is to develop all the context around the data, analyze it in that context, and present it to your users in the language and concepts they already know.TimescaleDB can help you do that, by giving you more time to focus on end-user value, and less time focusing on things like “Can I connect tool x to my store?” or “How am I going to scale this thing?”Other than those words of advice, there’s nothing that hasn’t already been covered in-depth by people in the PostgreSQL community that are FAR smarter than I am :).We’d like to give a big thank you to Nathan and the Conserv team for sharing their story and,more importantly, for their commitment to helping collections’ keep history alive. As big museum fans, we’re honored to play a part in the tech stack that powers their intuitive, easy-to-use UI and robust analytics 💛We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@lacey butler), and we’ll go from there.Additionally, if you’re looking for more ways to get involved and show your expertise, check out theTimescale Heroesprogram.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-conserv-safeguards-history-building-an-environmental-monitoring-and-preventative-analytics-iot-platform/
2020-10-05T17:38:11.000Z,How FlightAware Fuels Flight Prediction Models for Global Travelers With TimescaleDB and Grafana,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Caroline Rodewig, Senior Software Engineer and Predict Crew Lead at FlightAware,  joins us to share how they’ve architected a monitoring system that allows them to power real-time flight predictions, analyze prediction performance, and continuously improve their models.FlightAwareis the world's largestflight tracking and data platform; we fuse hundreds of global data sources to produce an accurate, consistent view of flights around the world. We make this data available to users through web and mobile applications, as well as different APIs.Our customers cover a number of different segments, including:Travelers / aviation enthusiastswho use our website and mobile apps to track flights (e.g., using our “where’s my flight?” program).Business aviation providers(such asFixed Base Operatorsoraircraft operators) who use flight-tracking data and custom reporting to support their businesses.Airlinesthat use flight-tracking data or our predictive applications to operate more efficiently.Editor’s Note: for more information about FlightAware’s products (and ways to harness its data infrastructure), check outthis overview. Want to build your own flight tracking receiver and ground station? SeeFlightAware’s PiAware tutorial.About the teamThe Predictive Technologiescrewis responsible for FlightAware's predictive applications, which as a whole are called ""FlightAware Foresight."" At the moment, our small-but-mighty team is made up of only three people: our project managerJames Parkman, software engineer Andrew Brooks, and myself. We each wear many different hats; a day's work can cover anything from Tier 2 customer support to R&D, and everything in between.A former crew member, Diorge Tavares, wrote acool articleabout his experience as a site reliability engineer embedded in the Predict crew. He helped us design infrastructure and led our foray into cloud computing; now that our team is more established, he’s moved back to the FlightAware Systems team full-time.About the projectOur team's chief project is predicting flight arrival times, or ETAs; we predict both landing (EON) and gate arrival (EIN) times. And, ultimately, we need to monitor, visualize, and alarm on thequalityof those predictions. This is where TimescaleDB fits in.Not only should we track how our prediction error changes over the course of each flight, we also need to track how our error changes over months - or years! - to ensure we're continually improving our predictions. Our predictive models can have short bursts of inaccuracy - like failing to anticipate the impact of a huge storm - but they can also drift slowly over time as real-life behaviors change.As an example of the type of data we extract, the below is our ""Worst Flights"" dashboard, which we use for QA. (Looking through outliers is an easy way to spot bugs.) The rightmost column compares our error to third-parties', so we can see how we're doing relative to the rest of the industry.Our Grafana dashboard for tracking ""Worst Flights"" and our prediction quality vs. other data sourcesBut, we also go deep into specific flights, like the below ""Single Flight"" dashboard view.This is useful for debugging, as it gives a detailed picture of how our predictions changed over the course of asingleflight.Our Grafana dashboard for debugging and assessing our prediction quality at the individual flight levelChoosing (and using) TimescaleDBWe tested out several different monitoring setups before settling on TimescaleDB and Grafana. We recently published ablog postdetailing our quest for a monitoring system, which I’ve summarized below.First, we considered usingZabbix; it's widely used at FlightAware, where most software reports into Zabbix in one way or another. However, we quickly realized that Zabbix was not the tool for the job – our Systems crew had serious doubts that Zabbix would be able to handle the load of all the metrics we wanted to track:We make predictions for around 75,000 flights per day; if we only stored two error values per flight (much fewer than we wanted), it would require making 100 inserts per minute.After ruling out Zabbix, I started looking atGrafanaas a visualization and alerting tool, and it seemed to have all the capabilities we needed. For my database backend, I first pickedPrometheus, because it was near the top of Grafana's ""supported databases"" list and its built-in visualization capabilities seemed promising for rapid development.I didn't know much about time-series databases, and, while Prometheus is a good fit for some data, it really didn't fit mine well:No JOINs. My only prior database experience was with PostgreSQL, and it didn't occur to me that some databases just wouldn't support JOINs. While wecouldhave worked around this issue by inserting specific, already-joined error metrics, this would have limited the flexibility and ""query-a-bility"" of the data.Number of labels to store. At the bare minimum, we wanted to store EON and EIN predictions for 600 airports, at least 10 times throughout each flight. This works out to 12,000 different label combinations, each stored as a time series – whichPrometheus is not currently designed to handle.And, that’s when I found TimescaleDB. A number of factors went into our decision to use TimescaleDB, but here are the top four:Excellent performance.This article comparing TimescaleDB vs. PostgreSQL performancereally impressed me. Getting consistent performance, despite the number of rows in the table, was critical to our goal of storing performance data over several years.Institutional knowledge.FlightAware uses PostgreSQL in a vast number of applications, so there was already a lot of institutional knowledge and comfort with SQL.Impressive documentation.I have yet to have an issue or question that wasn't discussed and answered in the docs. Plus, it was trivial to test out – I love one-line docker start-up commands (seeTimescaleDB Docker Installation instructions).Grafana support.I was pretty confident that I wanted to use Grafana to visualize our data and power our dashboards, so this was a potential dealbreaker.We use several Grafana dashboards, like this one, to view detailed performance over time (average error trends over one or more airports)Editor’s Note: To learn more about TimescaleDB and Grafana,see our Grafana tutorials(5 step-by-step guides for building visualizations, using variables, setting up alerts, and more) andGrafana how-to blog posts.To see how to use TimescaleDB to perform time-series forecasting and analysis,check out our time-series forecasting tutorial(includes two forecasting methods, best practices, and sample queries).Current deployment & use casesOur architecture is pretty simple (see diagram below). We run a copy of this setup in several environments: production, production hot-standby, staging, and test. Each environment has its own predictions database, which allows us to compare our predictions in staging to those in production and validate changesbeforethey get released.⭐Pro tip:we periodically sync Grafana configurations from production to each of the other environments, which reduces the manual work involved in updating dashboards across instances.FlightAware Predict team's system architecture, which uses custom Python programs, Docker, Grafana, and TimescaleDBAfter some trial and error, we’ve set up our TimescaleDB schema as follows:(1) Short term (1 week) tables for arrivals, our own predictions, and third-party predictions.The predict-assessor program reads our flight data feed, extracts ETA predictions and arrival times, and inserts them into the database. For scale, the arrivals table typically contains 500k rows, and the predictions tables each contain 5M rows.Each table is chunked: arrivals by arrival time and predictions by the time the prediction was made.We use archiving functions to copy some data into long-term storage, andadrop_chunkspolicyto ensure that rows older than one week are dropped to prevent unlimited table growth.(2) Long term (permanent) table for prediction and prediction-error data.Archiving functions move data to the long term table by joining the short terms tables together. They also ""threshold"" the data to reduce verbosity, by only storing predictions at predetermined intervals; i.e., predictions that were present 1 and 2 hours before arrival are migrated to long-term tables, but intermediate predictions (i.e., at 1.5 hours before arrival) are not kept.Between the join and the threshold, the archiving process reduces the average number rows per flight from25(across 3 short-term tables) to6!We haven’t enabled adrop_chunkspolicy on this table as of now; after ~9 months of running this setup, our database file is pretty manageable at 54GB. If we start having space issues, we'd opt to store fewer predictions per flight rather than lose any year-over-year historical data.Biggest ""Aha!"" momentContinuous aggregates are what well and truly sold me on TimescaleDB. We went from 6.4 seconds to execute a query to 30ms.Yes, milliseconds.I was embarrassingly late to the party when it comes to continuous aggregates. When I first set up our database, every query was fast because the database was small. However, as we added data over time, some queries slowed down significantly.The biggest offender was a query on our KPIs dashboard, visualized in Grafana below. This graph gives us a birds-eye view of error over time. The blue line represents the average error for all airports at a certain time before arrival; the red line shows the number of flights per day. (You can see the huge traffic drop when airlines stopped flights in March, due to the COVID-19 pandemic.)Our KPI dashboard includes various metrics, including our average error rate and total flights per day across all airportsBeforelearning about continuous aggregates, the query to extract this data looked like this:SELECT
  time_bucket('1 day', arr_time) AS ""time"",
  AVG(get_error(prediction_fa, arr_time)) AS on_error,
  count(*) AS on_count
FROM prediction_history
WHERE 
  time_out = '02:00:00' AND 
  arr_time BETWEEN '2020-03-01' AND '2020-09-05'
GROUP BY 1
ORDER BY 1It took 6.4 seconds and aggregated 1.6M rows, from a table of 147M rows.For what the query was doing, this runtime wasn't too bad – the table was chunked byarr_time, which the query planner could take advantage of.I considered adding indexes to make the query faster, but wasn't convinced they would help much and was concerned about the resulting performance penalties for inserts.I also considered creating a materialized view to aggregate the data and writing a cron job to regularly refresh it...but that seemed like a hassle, and after all, I could wait 10 seconds for something to load 🤷‍♀️.Then, I discovered TimescaleDB's continuous aggregations! For the unfamiliar, they basically implement that regularly-refreshing materialized view idea, but in a far smarter way and with a bunch of cool extra features.Here's the view for the continuous aggregate:CREATE VIEW error_by_time_out
WITH (timescaledb.continuous) AS
  SELECT
    time_out,
    time_bucket(INTERVAL '1 hour', arr_time) AS bucket,
    AVG(get_error(prediction_fa, arr_time)) AS avg_error,
    COUNT(*) AS count
  FROM prediction_history
  GROUP BY time_out, bucket;The new data extraction query is a little bit harder to parse, because the error needs to be aggregated across continuous aggregate buckets:SELECT
  time_bucket('1 day', bucket) AS ""time"",
  SUM(avg_error * count) / SUM(count) AS error,
  SUM(count) AS count
FROM error_by_time_out
WHERE 
  time_out = '02:00:00' AND 
  bucket BETWEEN '2020-03-01' AND '2020-09-05'
GROUP BY 1
ORDER BY 1...and I'll let you guess how long it takes....30ms.Yes, milliseconds. We went from 6.4 seconds to execute the query to 30ms.On top of that, unlike in a classic materialized view, the whole view doesn't have to be recalculated every time it needs to be updated -just the parts that have changed.This means refreshes are lightning fast too.Continuous aggregates are what well and truly sold me on TimescaleDB.The amazing developers at Timescale recently made continuous aggregates even better through ""real-time"" aggregates. These will automatically fill in data between the last view refresh and real-time when they're queried, so you always get the most up-to-date data possible. Unfortunately, our database is a few versions behind so we're not using real-time aggregates yet, but I can't wait to upgrade and start using them.Editor’s Note: To learn more about real-time aggregates and how they work, see our“Ensuring up-to-date results with Real-Time Aggregations” blog and mini-tutorial(includes benchmarks, example scenarios, and resources to get started).Getting started advice & resourcesIn addition to the documentation I’ve linked throughout this post, I'd recommend doing what I did: readingthe TimescaleDB docs, spinning up a test database, and going to town.And, after a few months of use, make sure to go back and read the docs again – you'll discover all sorts of new things to try to make your database even faster (looking at youtimescaledb-tune)!Editor’s Note: If you’d like to follow Caroline’s advice and start testing TimescaleDB for yourself,Timescale Cloudis the fastest way to get up and running - 100% free for 30 days, no credit card required. You can see self-managed and other hosted optionshere.To learn more about timescale-tune,see our Configuring TimescaleDB documentation.We’d like to thank Caroline and the FlightAware team for sharing their story, as well as for their work to make accurate, reliable flight data available to travelers, aviation enthusiasts, and operators everywhere. We’re big fans of FlightAware at Team Timescale, and we’re honored to have them as members of our community!We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@lacey butler), and we’ll go from there.Additionally, if you’re looking for more ways to get involved and show your expertise, check out theTimescale Heroesprogram.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-flightaware-fuels-flight-prediction-models-with-timescaledb-and-grafana/
2022-03-01T13:57:33.000Z,CloudQuery on Using PostgreSQL for Cloud Assets Visibility,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Ron Eliahu, CTO and co-founder at CloudQuery, joins us to share how they transform data about cloud assets into PostgreSQL tables to give developers visibility into the health of cloud infrastructure. Thanks toTimescaleDB, CloudQuery users manage their data more transparently while maintaining scalability.CloudQueryis an open-source cloud asset inventory powered by SQL. CloudQuery extracts, transforms, and loads cloud assets intonormalizedPostgreSQL tables, enabling developers to assess, audit, and monitor the configurations of their cloud assets.Cloud asset inventory is a key component to solve various challenges in the cloud:Cloud Infrastructure Monitoring, Visibility, and Search: Give developers, SREs, DevOps, and security engineers a streamlined way to gain visibility and perform a wide range of tasks. These tasks include security analytics, fleet management auditing, governance, and cost.Security & Compliance: Turn security and compliance tasks to data problems and solve them with the best tools and practices in DataOps. UseCloudQuery Policiesto codify, version control, and automate security and compliance rules with SQL.Infrastructure as Code (IaC) Drift Detection: CloudQuery leverages its asset inventory to quickly detect drift against IaC (Terraform, more to come) which you can run both in the CI and locally.📖Read CloudQuery´sannouncement blogabout releasing CloudQuery History in Alpha and adding support for TimescaleDB.About the teamWe started CloudQuery a year ago, to solve the cloud asset visibility problem, and quickly gained traction. We are currently a small but mighty team of open-source and cloud security enthusiasts, spread all around the world!A little about myselfRon Eliahu, I am the CTO and co-founder at CloudQuery, I am an engineer at heart, I love building open source projects and working with anything database-related.About the projectQueryable cloud asset inventory is key in solving a lot of core challenges in the cloud such as security, compliance, search, cost, and IaC drift detection. That is why we started CloudQuery and followed a few key decisions:PostgreSQL- The most used database in the world with a huge ecosystem of business intelligence and visualization tools.Open-source- To cover a huge amount of API and cloud providers we decided to make this open-source where everyone can contribute without being blocked by a vendor.Pluggable Architecture with CloudQuery SDK- Writing plugins, extracting configuration data, transforming it, and loading it to PostgreSQL requires a lot of boilerplate code. To scale this and improve the developer’s experience both internally and externally we releasedCloudQuery SDK.Normalized PostgreSQL tables inDataGripcontaining data about cloud assets from Microsoft AzureAnother key observation and requirement that we saw early on for a cloud asset inventory is the ability to not only query the current state but also go back in time. This is super useful for tasks such as forensics, post-mortems, compliance, and more.This feature required us to maintain historical snapshots in PostgreSQL and we started to look out for a solution, which was quite a journey for us.Choosing (and using!) TimescaleDBFirst attempt: PostgreSQL PartitioningWith some good experience ofPostgreSQLunder our belt, the first thing we tried isPostgreSQL partitioningPretty quickly it turned out to be not as easy as expected, hard to maintain and manage, lacking easily creatable retention policies, and bucketing queries. Given our philosophy is to integrate with best-in-class tools and focus our development efforts on our core business use-cases we started looking for an alternative solution.✨Editor’s Note:For more comparisons and benchmarks, seehow TimescaleDB compares to InfluxDB, MongoDB, AWS Timestream, vanilla PostgreSQL, and other time-series database alternativeson various vectors, from performance and ecosystem to query language and beyond.Second attempt: TimescaleDBGiven CloudQuery uses PostgreSQL under the hood, supporting historical snapshots in a scalable way usually involves using partitioning. TimescaleDB´screate_hyperfunctionsfeature allows us to do just that in a simple and effective way giving our users a transparent and automated way to manage their data while still maintaining scalability.Current CloudQuery architecture diagramCloudQuery transforms cloud resources into tables, some of these resources have a complex data structure, and are split into multiple relational tables, some of which are hypertables. In order to retain data integrity, we use foreign key relationships (withON DELETE CASCADE) to the root resource table. With these foreign keys in place, if a reference to a cloud resource is deleted (for instance a S3 bucket), the downstream data is removed.While TimescaleDB hypertablesdosupport using foreign keys that reference a regular PostgreSQL table, hypertable cannot be thesource reference of a foreign keyIn our case, some of our ""root"" reference tables are hypertables which meant that we had to come up with another way to do the cascading deletes to retain data integrity.A common alternative is to create trigger functions that will cause a delete on the relation table if a row is deleted in the parent table, the issue here is that some resources in CloudQuery can have three or more levels of relations and we didn’t want to create many queries to solve this, so we came up with the following functions to easily create the deletion cascade.First, we wanted a trigger function that will delete our relational table. We used trigger arguments to pass the relation table name its foreign key name so we can delete the data in the relation table. Full code is availablehere.CREATE OR REPLACE FUNCTION history.cascade_delete()
					RETURNS trigger
					LANGUAGE 'plpgsql'
					COST 100
					VOLATILE NOT LEAKPROOF
				AS $BODY$
				BEGIN
					BEGIN
						IF (TG_OP = 'DELETE') THEN
							EXECUTE format('DELETE FROM history.%I where %I = %L AND cq_fetch_date = %L', TG_ARGV[0], TG_ARGV[1], OLD.cq_id, OLD.cq_fetch_date);
							RETURN OLD;
						END IF;
						RETURN NULL; -- result is ignored since this is an AFTER trigger
					END;
				END;
				$BODY$;Then, we call the create trigger function on the root table and pass these arguments to the child. Full code is availablehere.CREATE OR REPLACE FUNCTION history.build_trigger(_table_name text, _child_table_name text, _parent_id text)
					RETURNS integer
					LANGUAGE 'plpgsql'
					COST 100
					VOLATILE PARALLEL UNSAFE
				AS $BODY$
				BEGIN
					IF NOT EXISTS ( SELECT 1 FROM pg_trigger WHERE tgname = _child_table_name )  then
					EXECUTE format(
						'CREATE TRIGGER %I BEFORE DELETE ON history.%I FOR EACH ROW EXECUTE PROCEDURE history.cascade_delete(%s, %s)'::text,
						_child_table_name, _table_name, _child_table_name, _parent_id);
					return 0;
					ELSE
						return 1;
					END IF;
				END;
				$BODY$;To sum it all up, we built two generic SQL functions to make sure all our hypertables and relational hypertables data get deleted if the root table has any data removed.Future plansCompliance overtime is a common request so we are working on integrating the results ofCloudQuery Policieswith TimescaleDB so you can monitor and visualize compliance with TimescaleDB and Grafana.Getting started advice & resourcesBefore you jump into implementing your own partition strategy, definitely give TimescaleDB a try. It can save you a lot of development time and make your product more robust.The Timescale documentationis a great place to start.🗞️Subscribe to our newsletter atcloudquery.ioand join ourDiscordto hear about our upcoming and latest features.We’d like to thank Ron and all folks at the CloudQuery team for sharing their story, as well as for their work to transform complex and scattered cloud assets data into structured and easily accessible tables enabling developers to monitor their cloud inventory.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Lucie Šimečková), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/cloudquery-on-using-postgresql-for-cloud-assets-visibility/
2022-06-09T13:13:28.000Z,Processing and Protecting Hundreds of Terabytes of Blockchain Data: Zondax’s Story,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, Ezequiel Raynaudo, data team leader at Zondax, explains how the software company consumes and analyzes hundreds of terabytes of blockchain data using TimescaleDB to create complete backend tech solutions.About the CompanyZondaxis a growing international team of more than 30 experienced professionals united by a passion for technology and innovation. Our end-to-end software solutions are widely used by many exchanges, hardware wallets, privacy coins, and decentralized finance (DeFi) protocols. Security is one of the highest priorities in our work, and we aim to provide the safest and most efficient solutions for our clients.Founded in 2018, Zondax has been consideredthe leader in the industry for Ledger apps development, with more than 45 applications built to date and more near production. Since its inception, our team has been building and delivering high-quality backend tech solutions for leading blockchain projects of prominent clients, such as Cosmos, Tezos, Zcash, Filecoin, Polkadot, ICP, Centrifuge, and more.“The database is a critical part of the foundation that supports our software services for leading blockchain projects, financial services, and prominent ecosystems in the blockchain space”We put a lot of emphasis on providing maximum safety and efficiency for the ecosystem. Our services can be categorized into security, data indexing,  integration, and protocol engineering. Each category has designated project leaders that take the initiative in managing the projects with well-experienced engineers.About the TeamMy team (which currently has two people but is looking to grow by the end of the year) manages the blockchain data in various projects and ensures the tools needed for the projects are well-maintained. For example, we pay close attention to the dynamic sets of different blockchains and create advanced mathematical models for discovering insightful information via data analytics.The database is a critical part of the foundation that supports our software services for leading blockchain projects, financial services, and prominent ecosystems in the blockchain space. In conclusion, we take serious steps to ensure the work of processing blockchain data remains effective, and that the quality of the results meets Zondax's high standards.We welcome professionals from different cultures, backgrounds, fields of experience, and mindsets. New ideas are always encouraged, and the diversity within the team has been helping us to identify various potential advancements we can make in leading blockchain projects.At the same time, our efforts in experimenting and searching for creative and efficient solutions led us to pay close attention to the latest technologies and innovations that we immediately incorporate into our work. We never get bored at Zondax!Since the Covid-19 pandemic, many companies and teams have switched to remote work temporarily or permanently, and for Zondax this is familiar ground. We adopted a culture of remote work from the start, which has been rewarding and encouraging as team members from around the globe often spark fun and constructive discussions despite nuanced cultural differences. And in terms of quality of work, the tools and platforms we have been using accommodate the needs for smooth communication and effective collaboration within different teams.About the ProjectZondax provides software services to various customers, including leading projects in the blockchain ecosystem and financial services. We consume and analyze blockchain data in numerous different ways and for multiple purposes:As input for other services and ecosystems provided by Zondax and for other third partiesTo apply data science and get value in the form of insights by using mathematical models for different blockchain dynamic sets of variablesFinancial servicesBlockchain data backupsThe minimal unit of division of a blockchain, a.k.a. a “block” on most networks, contains a timestamp field among several other sub-structures. It allows you to define blocks at a specific point in time throughout the blockchain history. So having a database engine that can leverage that property is a go-to option.Choosing (and Using!) TimescaleDB✨Editor’s Note:Want to know more about optimizing queries on hypertables with thousands of partitions using TimescaleDB?Check out this blog post.Our first encounter with TimescaleDB was througha blog post about database optimizations. We decided to go ahead and install it on our infrastructure since it’s built on top of PostgreSQL, and our main code and queries didn’t require to be updated. After installing the corresponding helm chart on our infrastructure, we decided to try it.The first tests denoted a substantial increase in performance when writing or querying the database, with no optimizations at all and without using hypertables. Those results encouraged us to keep digging into TimescaleDB’s configurations and optimizations, such as usingtimescaledb-tuneand converting critical tables to hypertables.“If Timescale didn’t exist, we would have a problem and might need to wait a couple of weeks to process a few dozens of terabytes rather than waiting only 1-2 days”Long story short, we went from having to wait a couple of weeks to process a few dozens of terabytes to only 1-2 days. Among the top benefits of TimescaleDB, I would highlight having the best possible write and read performance. It is a critical part of our ecosystem because it helps us provide fast and responsive services in real time. Using TimescaleDB also allows our software to stay synced with the blockchain's nodes, which is one of the most significant acknowledged advantages of our software and services. Last but not least, we also use TimescaleDB for blockchain data backups to protect data integrity.Before finding out about TimescaleDB, we first used custom PostgreSQL modifications like indexing strategies and high-availability setups. Also, our team did some benchmarking using NoSQL databases like MongoDB, but with no substantial improvements on the write/read speeds that we needed.If Timescale didn’t exist, we would have a problem and might need to wait a couple of weeks to process a few dozens of terabytes rather than waiting only 1-2 days.We are glad that we chose Timescale and proud of the work that has been expedited and achieved. For example, despite many challenges, we didn't give up on experimenting with new approaches to process a tremendous amount of blockchain data. Instead, we continued exploring new ideas and tools until we eventually started using TimescaleDB, which drastically shortened the time to process data and accelerated our progress in delivering quality results for the projects.Current Deployment & Future Plans✨Editor’s Note:Read how you canadd TimescaleDB to your Kubernetes deployment strategyquickly and easily.We deploy TimescaleDB using a custom helm chart that fits our infrastructure needs. As far as programming languages, we mainly use Golang to interact with TimescaleDB; and Hasura as the main query engine for external users.Advice & Resources✨Editor’s Note:Want to learn how to create a hypertable using TimescaleDB?Check out our documentation on hypertables and chunks.I’d recommend reading the blog postson how to get a working deployment,hypertables, andTimescale vs. vanilla Postgres's performance using the same queries.A wise man (Yoda) once said, ""You must unlearn what you have learned."" It is inevitable to encounter countless challenges when developing a scalable database strategy, but staying curious and willing to explore new solutions with caution can sometimes be rewarding.We’d like to thank Ezequiel and all of the folks at Zondax for sharing their story and efforts in finding a solution to process enormous amounts of data to build backend solutions for blockchain projects.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/processing-and-protecting-hundreds-of-terabytes-of-blockchain-data-zondaxs-story/
2022-10-20T13:00:00.000Z,How United Manufacturing Hub Is Introducing Open Source to Manufacturing and Using Time-Series Data for Predictive Maintenance,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition, we speak with Jeremy Theocharis, co-founder and CTO ofUnited Manufacturing Hub, about how they are bringing open source to the world of manufacturing by combining information and operational tools and technologies in an open-source Helm chart for Kubernetes.The team uses TimescaleDB to store both relational and time-series data coming fromMQTTandKafkaand then visualizes it usingGrafanaand their own REST API. With the data, they can prevent and predict maintenance issues, analyze and optimize production losses such as changeovers or micro-stops and reduce resource consumption, and much more.About the CompanyWe believe that open source is the future: we live in a world where 100 percent of all supercomputers useLinux, 95 percent of all public cloud providers useKubernetes, and the Mars Helicopter, Ingenuity, usesf-prime.We are confident that open source will also find its place in manufacturing, and we were among the first to use it in this field, making us the experts.Our story goes back to 2016, when we had the pain of integrating and maintaining various costly Industrial IoT solutions (IIoT). Existing vendors focused only on the end result, which resulted in large-scale IIoT projects failing because they did not address the real challenges.After suffering for years, in 2021, we were fed up and decided to do something about it. Since then, all our products and services have been focused on efficiently integrating and operating large-scale IIoT infrastructures.About the TeamWe are a team of 11 people with different backgrounds, from mechanical engineering to business administration to cybersecurity.Me, personally? I am an IT nerd that learned programming at 14 and then studied Mechanical Engineering and Business Administration atRWTH Aachenin Germany. I did my internship first as a technical project manager in theDigital Capability Center Aachen, where I was responsible for the technical integration of various Industrial IoT solutions. I later did another internship at McKinsey & Company in Singapore and decided I needed a solid technical part in my future profession.I started my own business as a system integrator in the Industrial IoT sector, metChristianandAlex, and founded theUMH Systems GmbHtogether in 2021.About the ProjectUnited Manufacturing Hub's architectureTheUnited Manufacturing Hub(UMH) is an open-source Helm chart for Kubernetes. It combines state-of-the-art information and operational tools (IT/OT) and technologies, bringing them into the engineer's hands.I assume most of the readers here will come from traditional IT, so let me explain the world of manufacturing, especially OT,  first, as it will help you understand our usage of TimescaleDB.OT means Operational Technology. OT is the hardware and software to manage, monitor, and control industrial operations like production machines. Due to different requirements, OT has its own ecosystems.OT comes originally from the field of electronics, and this background is still evident today. For example, the computer that controls the production machine is called aPLC (Programmable Logic Controller). It runs some peculiar flavors of Windows and Linux, which are completely hidden from the system's programmer. The programmer of the PLC will use programming languages likeladder logic, which is like drawing electrical schematics.Because OT is an entirely different world, it is pretty hard to integrate it with the traditional IT world. During system integration, we felt all these pains and decided to develop a tool that allows an easy combination between both fields—the United Manufacturing Hub (UMH).With UMH, one can now easily extract data from the shopfloor, from the PLC tovarious IO-link compatible or analog (4-20mA / 0-10V) sensorstobarcodereaderand different manufacturing execution or enterprise resource planning systems. Using aUnified Namespacebased onMQTTandKafka, the data is aggregated and can then be contextualized through tools likeNode-RED.From there on, the processed data is stored automatically in a TimescaleDB running either in the cloud or on-premise. To visualize the data, we useGrafanawith our ownREST APIfor manufacturing specific logics (also calledfactoryinsight) and our own Grafana data source.Choosing (and Using!) TimescaleDBManufacturing data is mainly relational: orders, products, production plans, and shifts are good examples of this. However, due to the growth of analytics, time-series data gets more and more important, e.g., for preventive or predictive maintenance.✨Editor's Note:Want to learn more about time-series forecasting?Check out this blog post.During one of those earlier system integrator projects, I realized that we needed a time-series database and a relational one.Due to the strong marketing, we chose InfluxDB at first. We did not scan vendors; we just started with whatever we knew from home automation. It sounded perfect: a beautiful user interface, continuous queries to process data, etc.We wanted to process raw sensor data, e.g., converting the distance of a light barrier into the machine status (running/not running). We also needed to store shifts, orders, and products and model the data. We did that via InfluxDB as well.The project was a nightmare. To be fair, InfluxDB was not its main driver, but it definitely was in the top five. Modeling relational data into a time-series database is a bad idea. The continuous queries were failing too often without even throwing error messages. The system could not handle the data buffered somewhere in the system and arrived late.“The stability of TimescaleDB allows us to focus on developing our microservices instead of running around fixing breaking API changes”Additionally, Flux as a query language is comparatively new and not as easy to work with as SQL. It quickly reached the point where we had to implement Python scripts to process data because Flux had reached its limits in use cases that would work seamlessly using SQL. So we felt like InfluxDB was putting unnecessary obstacles in our way.We even wrote a blog article aboutwhy we chose TimescaleDB over InfluxDB for the field of Industrial IoT.One of the main factors for us to use TimescaleDB as our database is the reliability and fault tolerance [the ability of a system to continue operating properly in case of failure] it offers to our stack. Since PostgreSQL has been in development for over 25 years, it is already very robust.“The reliability also manifests in the ease of horizontal scaling across multiple servers, which we are very interested in”The stability of TimescaleDB allows us to focus on developing our microservices instead of running around fixing breaking API changes, which newer, less stable databases like InfluxDB have shown to bring forth.The reliability also manifests in the ease of horizontal scaling across multiple servers, which we are very interested in.Being based on SQL was also a factor for us as SQL is the most well-known query language for relational databases—making working with it much easier. Almost any possible problem is already documented and solved somewhere on the Internet.Now, TimescaleDB is used in our stack as our main database to store the data coming in via MQTT/Kafka. We are storing (among others) machine states, product states, orders, worker shifts, and sensor data. Some are relational; some are time-series.If TimescaleDB didn’t exist, we probably would have to employ a PostgreSQL-based relational database system in addition to InfluxDB for time-series data. That would mean a lot of additional effort as we would have to manage two separate databases and the creation of datasets that span the two. This would also make the system more prone to errors as we would have to employ multiple querying languages.Current Deployment & Future PlansAs I mentioned, the United Manufacturing Hub is an open-source Helm chart for Kubernetes, which combines state-of-the-art IT/OT tools and technologies and brings them into the hands of the engineer.This allows us to standardize the IT/OT infrastructure across customers and makes the entire infrastructure easy to integrate and maintain.We typically deploy it on the edge and on-premise usingk3sas light Kubernetes. In the cloud,  we use managed Kubernetes services likeAKS. If the customer is scaling out and okay with using the cloud, we recommend services likeTimescale.We are using TimescaleDB with MQTT, Kafka, and Grafana. We have microservices to subscribe to the messages from the message brokers MQTT and Kafka and insert the data into TimescaleDB, as well as a microservice that reads out data and processes it before sending it to a Grafana plugin, which then allows for visualization.✨Editor’s Note:Learn how you can improve your Grafana performance using downsampling in TimescaleDB.RoadmapWe are currently positioning the United Manufacturing Hub with TimescaleDB as an open-source Historian. To achieve this, we are currently developing a user interface on top of the UMH so that OT engineers can use it and IT can still maintain it.We can recommend our blog articlefor a good comparison between Historians and Open-Source databases.Furthermore, we are developing a Management Console on top of the Helm chart, which makes a lot of the typical operation tasks (monitoring, logging, changing the configuration, etc.) easily accessible for the OT engineer, reducing the workload of maintaining all the edge devices, servers, and so on for the IT person.Advice & ResourcesFor manufacturing, we recommend the previously mentioned blog articles and the official TimescaleDB documentation. For data models and data ingestions from MQTT and Kafka into TimescaleDB, we can also recommend looking at the United Manufacturing Hub source code (or using it directly).One last piece of advice: I can strongly recommend the bookDesigning Data-Intensive Applicationsby Martin Kleppmann. It really helped me understand the fundamental principles in designing large-scale architectures so you can join discussions on the technical level. It explains the fundamental choices behind databases (from log-based approaches over WAL to binary trees) and the problems and solutions for distributed systems.We’d like to thank Jeremy Theocharis and the folks and United Manufacturing Hub for sharing their story on how they are using TimescaleDB to store their data, and why they chose us over other databases.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Ana Tavares), and we’ll go from there.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-united-manufacturing-hub-is-introducing-open-source-to-manufacturing-and-using-time-series-data-for-predictive-maintenance/
2021-07-26T14:42:35.000Z,How METER Group Brings a Data-Driven Approach to the Cannabis Production Industry,"This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Paolo Bergantino, Director of Software for the Horticulture business unit at METER Group, joins us to share how they make data accessible to their customers so that they can maximize their cannabis yield and increase efficiency and consistency between grows.AROYAis the leading cannabis production platform servicing the U.S. market today. AROYA is part ofMETER Group, a scientific instrumentation company with 30+ years of expertise in developing sensors for the agriculture and food industries. We have taken this technical expertise and applied it to the cannabis market, developing a platform that allows growers to grow more efficiently and increase their yields – and to do so consistently and at scale.About the teamMy name isPaolo Bergantino. I have about 15 years of experience developing web applications in various stacks, and I have spent the last four here at METER Group. Currently, I am the Director of Software for the Horticulture business unit, which is in charge of the development and infrastructure of the AROYA software platform. My direct team consists of about ten engineers, 3 QA engineers, and a UI/UX Designer. (We’re also hiring!)About the projectAROYA is built as a React Single-Page App (SPA) that communicates with a Django/DRF back-end. In addition to usingTimescale Cloudfor our database, we use AWS services such as EC2+ELB for our app and workers,ElastiCache for Redis,S3for various tasks,AWS IoT/SQSfor handling packets from our sensors, and some other services here and there.✨Editor’s Note:""Timescale Cloud"" is known as ""Managed Service for TimescaleDB"" as of September 2021.As I previously mentioned, AROYA was born out of our desire to build a system that leveraged our superior sensor technology in an industry that needed such a system. Cannabis worked out great in this respect, as the current legalization movement throughout the U.S. has resulted in a lot of disruption in the space.The more we spoke to growers, the more we were struck by how much mythology there was in growing cannabis and by how little science was being applied by relatively large operations. As a company with deeply scientific roots, we found it to be a perfect match and an area where we could bring some of our knowledge to the forefront.We ultimately believe the only survivors in the space are those who can use data-driven approaches to their cultivation to maximize their yield and increase efficiency and consistency between grows.As part of the AROYA platform, we developed a wireless module (called a “nose”) that could be attached to our sensors. Using Bluetooth Low Energy (BLE) for low power consumption and attaching a solar panel to take advantage of the lights in a grow room, the module can run indefinitely without charging.The AROYA nose in its natural habitat (aroya.ioInstagram)The most critical sensor we attach to this nose is called the TEROS 12, the three-pronged sensor pictured below. It can be installed into any growing medium (like rockwool, coconut coir, soil, or mixes like perlite, pumice, or peat moss) and give insights into the temperature, water content (WC), and electrical conductivity (EC) of the medium. Without getting too into the weeds (pardon the pun), WC and EC, in particular, are crucial in helping growers make informed irrigation decisions that will steer the plants into the right state and ultimately maximize their yield potential.The AROYA nose with a connected TEROS 12 sensor (aroya.ioInstagram)We also have an ATMOS 14 sensor for measuring the climate in the rooms anda whole suite of sensorsfor other use cases.An AROYA repeater with an ATMOS 14 sensor for measuring the climate (aroya.ioInstagram)AROYA’s core competency is collecting this data - e.g., EC, WC, soil temp, air temperature, etc. - and serving it to our clients in real-time (or, at least “real-time” for our purposes, as our typical sampling interval is 3 minutes).Growers typically split their growing rooms into irrigation zones. We encourage them to install statistically significant amounts of sensors into each room and its zones, so that AROYA gives them good and actionable feedback on the state of their room.  For example, there’s a concept in cultivation called ""crop steering"" that basically says that if you stress the plant in just the right way,  you can ""steer"" it into generative or vegetative states at will and drive it to squeeze every last bit of flower. How and when you do this is crucial to doing it properly.Our data allows growers to dial in their irrigation strategy, so they can hit their target ""dryback"" for the plant (this is more or less the difference between the water content at the end of irrigation until the next irrigation event). Optimizing dryback is one of the biggest factors in making crop steering work, and it's basically impossible to do well without good data. (We provide lots of other data that helps growers make decisions, but this is one of the most important ones.)Graph showing electrical conductivity (EC) and water content (WC) data related to a room in AROYA.This can be even more important when multiple cultivars (“strains”) of cannabis are grown in the same room, as the differences between two cultivars regarding their needs and expectations can be pretty dramatic. For those unfamiliar with the field, an example might be that different cultivars ""drink"" water differently, and thus must be irrigated differently to achieve maximum yields. There are also ""stretchy"" cultivars that grow taller faster than ""stocky"" ones, and this also affects how they interact with the environment. AROYA not only helps in terms of sensing, but in documenting and helping understand these differences to improve future runs.The most important thing from collecting all this data is making it accessible to users via graphs and visualizations in an intuitive, reliable, and accurate way, so they can make informed decisions about their cultivation.We also have alerts and other logic that we apply to incoming data. These visualizations and business logic can happen at the sensor level, at the zone level, at the room level, or sometimes even at the facility level.A typical use case with AROYA might be that a user logs in to their dashboard to view sensor data for a room. Initially, they view charts aggregated to the zone level, but they may decide to dig deeper into a particular zone and view the individual sensors that make up that zone. Or, vice versa, they may want to pull out and view data averaged all the way up to the room. So, as we designed our solution, we needed to ensure we could get to (and provide) the data at the right aggregation level quickly.Choosing and using TimescaleDBThe initial solutionDuring the days of our closed alpha and beta of AROYA with early trial accounts (late 2017 through our official launch December 2019), the amount of data coming into the system was not significant. Our nose was still being developed (and hardware development is nice and slow), so we had to make due with some legacy data loggers that METER also produces. These data loggers only sampled every 5 minutes and, at best, reported every 15 minutes. We usedAWS’ RDS Aurora PostgreSQLservice and cobbled together a set of triggers and functions that partitioned our main readings table by each client facility – but no more. Because we have so many sensor models and data types we can collect, I chose to use anarrow data modelfor our main readings table.This overall setup worked well enough at first, but as we progressed from alpha to beta and our customer base grew, it became increasingly clear that it was not a long-term solution for our time series data needs. I could have expanded my self-managed system of triggers and functions and cobbled together additional partitions within a facility, but this did not seem ideal. There had to be a better way!I started looking into specific time-series solutions. I am a bit of a home automation aficionado, and I was already familiar with InfluxDB – butI didn’t wish to split my relational data and readings data or teach my team a new query language.TimescaleDB, being built on top of PostgreSQL, initially drew my attention: it “just worked” in every respect, I could expect it to, and I could use the same tools I was used to for it.At this point, however, I had a few reservations about some non-technical aspects of hosting TimescaleDB that prevented me from going full steam ahead with it.✨Editor’s Note:For more comparisons and benchmarks, seehow TimescaleDB compares to InfluxDB, MongoDB, AWS Timestream, and other time-series database alternativeson various vectors, from performance and ecosystem to query language and beyond.Applying a band-aid and setting a goalBefore this point, if I am perfectly truthful, I did not have any serious requirements or standards about what I considered to be the adequate quality of service for our application. I had a bit of an “I know it when I see it” attitude towards the whole thing.When we had a potential client walk away during a demo due to a particularly slow loading graph, I knew that we had a problem on our hands and that we needed something really solid for the long term.Still, at the time, we also needed something to get us by until we could perform a thorough evaluation of the available solutions and build something around that. At this point, I decided to stand aRedis clusterbetween RDS and our application which stored the last 30 days of sensor data (at all the aggregation levels required) as a Pandas dataframe. Any chart request coming in for data within the first 30 days - which accounted for something like 90% of our requests - would simply hit Redis. Anything longer would cobble together the answer using both Redis and querying the database.Performance for the 90% use case was adequate, but it was getting increasingly dreadful as more and more historical data piled up for anything that hit the database.At this point, I set the goalposts for what our new solution would need to meet:Any chart request, which is an integral part of AROYA, needs to take less than one second for the API to serve.The research and the first solutionWe looked at other databases at this point, InfluxDB was looked at again, we got in a beta of Timestream for AWS and looked at that. We even considered going NoSQL for the whole thing. We ran tests and benchmarks, created matrices of pros and cons, estimated costs, the whole shebang.Nothing compared favorably to what we were able to achieve with TimescaleDB.Ultimately,the feature that really caught our attention wascontinuous aggregatesin TimescaleDB. The way our logic works, more or less, we see the timeframe that the user is requesting and sample our data accordingly. In other words, if a user fetches three months worth of data, we would not send three months worth of raw data to be graphed to the front-end. Instead, we would bucket our data into appropriately sized buckets that would give us the right amount of data we want to display in the interface.Although it would require quite a few views, if we created continuous aggregates for every aggregation level and bucket size we cared about, and then directly queried the right aggregation/bucket combination (depending on the parameters requested), that should do it, right? The answer was a resoundingyes.The performance we were able to achieve using these views shattered the competition.Although I admit we were kind of “cheating” by precalculating the data, the point is that we could easily do it. Not only this, but when we ran load tests on our proposed infrastructure, we were blown away by how much more traffic we could support without any service degradation. We could also eliminate all the complicated infrastructure that our Redis layer required, which was quite a load off (literally and figuratively).Grafana dashboard for the internal team showing app server load average before and after deployment of initial TimescaleDB implementation.The Achilles’ heel of this solution, an astute reader may already notice, is that we were paying for this performance in disk space.I initially brushed this off as fair trade and moved on with my life.We foundTimescaleDB’s compressionto be as good as advertised, which gave us 90%+ space savings in our underlying hypertable,but our sizable collection of uncompressed continuous aggregates grew by the day (keep reading to learn why this is a “but”...).✨Editor’s Note: We’ve put together resources aboutcontinuous aggregatesandcompressionto help you get started.The “final” solutionAROYA has been on an amazing trajectory since launch, and our growth was evident in the months before and after we deployed our initial TimescaleDB implementation. Thousands upon thousands of sensors hitting the field was great for business – but bad for our disk space.Our monitoring told a good story of how long our chart requests were taking, as 95%+ of them were under 1 second, and virtually all were under 2 seconds. Still, within a few months of deployment, we needed to upgrade tiers in Timescale Cloud solely to keep up with our disk usage.✨Editor’s Note:""Timescale Cloud"" is known as ""Managed Service for TimescaleDB"" as of September 2021.We had adequate computing resources for our load, but 1 TB was no longer enough, so we doubled our total instance size to get another 1 TB. While everything was running smoothly, I felt a dark cloud overhead as our continuous aggregates grew and grew in size.The clock was ticking, and before we knew it, we were coming up on 2 TB of readings. So, we had to take action.We had attended a webinar hosted by Timescale and heard someone make a relatively off-hand comment about rolling their own compression for continuous aggregates. This planted a seed that was all we needed to get going.The plan was thus: first, after consulting with Timescale staff, we were alerted we had way too many bucket sizes. We could useTimescaleDB’s time_bucket functionsto do some of this on the fly without affecting performance or keeping as many continuous aggregates. That was an easy win.Next, we split each of our current continuous aggregates into three separate components:First, we kept the original continuous aggregate.Then, we leveraged theTimescaleDB job schedulerto move and compress chunks from the original continuous aggregate into ahypertablefor that specific bucket/aggregation view.Finally, we created a plain old view that UNIONed the two and made it a transparent change for our application.This allowed us to compress everything but the last week of all of our continuous aggregates, and the results were as good as we could have hoped for.The 1.83TB database was compressed into 700 GB.We were able to take our ~1.83 TB database and compress it down to 700 GB. Not only that, about 300 GB of that is log data that’s unrelated to our main reading pipeline.We will be migrating out this data soon, which gives us a vast amount of room to grow. (We think we can even move back the 1TB plan at this point, but have to test to ensure that compute doesn’t become an issue.) The rate of incrementation in disk usage was also massively slowed, which bodes well for this solution in the long term. What’s more, there was virtually no penalty for doing this in terms of performance for any of the metrics we monitor.Our monitoring shows how long sampling of chart requests takes to serve.Ultimately TimescaleDB had wins across the board for my team.Performance was going to be the driving force behind whatever we went with, and TimescaleDB has delivered that in spades.Current deployment & future plansWe currently ingest billions of readings every month using TimescaleDB and couldn’t be happier.Our data ingest and charting capabilities are two of the essential aspects of AROYA’s infrastructure.While the road to get here has been a huge learning experience, our current infrastructure is straightforward and performant, and we’ve been able to rely on it to work as expected and to do the right thing. I am not sure I can pay a bigger compliment than that.The current architecture diagramWe’ve recently gone live with our AROYA Analytics release, which is building upon what we’ve done to deliver deeper insights into the environment and the operations at the facilities using our service. Every step of the way, it’s been straightforward (and performant!) to calculate the metrics we need with our TimescaleDB setup.Getting started advice & resourcesI think it’s worth mentioning that there were many trade-offs and requirements that guided me to where AROYA is today with our use of TimescaleDB. Ultimately, my story is simply the set of decisions that led me to where we are now and people’s mileage may vary depending on their requirements.I am sure that the set of functionality offered means that, with a little bit of creativity, TimescaleDB can work for just about any time-series use case I can think of.The exercise we went through when iterating from our initial non-Timescale solution to Timescale was crucial to get me to be comfortable with that migration. Moving such a critical part of my infrastructure was scary, and it isstillscary.Monitoring everything you can, having redundancies, and being vigilant about any unexpected activity - even if it’s not something that may trigger an error - has helped us stay out of trouble.We have a bigGrafanadashboard on a TV in our office that displays various metrics and multiple times we’ve seen something odd and uncovered an issue that could have festered into something much more if we hadn’t dug into it right away. Finally, diligent load testing of the infrastructure and staging runs of any significant modifications have made our deployments a lot less stressful, since they instill quite a bit of confidence.✨ Editor’s Note:Check outGrafana 101 video seriesandGrafana tutorialsto learn everything from building awesome, interactive visualizations to setting up custom alerts, sharing dashboards with teammates, and solving common issues.I would like to give a big shout-out to Neil Parker, who is my right-hand man in anything relating to AROYA infrastructure and did virtually all of the actual work in getting many of these things set up and running. I would also like to thankMike FreedmanandPriscila Fletcherfrom Timescale, who have given us a great bit of time and information and helped us in our journey with TimescaleDB.We’d like to give a big thank you to Paolo and everyone at AROYA for sharing their story, as well as for their efforts to help transform the cannabis production industry, equipping growers with the data they need to improve their crops, make informed decisions, and beyond.We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Lucie Šimečková), and we’ll go from there.Additionally, if you’re looking for more ways to get involved and show your expertise, check out theTimescale Heroesprogram.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-meter-group-brings-a-data-driven-approach-to-the-cannabis-production-industry/
2021-12-17T14:40:33.000Z,"How to Build a Weather Station With Elixir, Nerves, and TimescaleDB","This is an installment of our “Community Member Spotlight” series, where we invite our customers to share their work, shining a light on their success and inspiring others with new ways to use technology to solve problems.In this edition,Alexander Koutmos, author of the Build a Weather Station with Elixir and Nerves book, joins us to share how he uses Grafana and TimescaleDB to store and visualize weather data collected from IoT sensors.About the teamThe bookBuild a Weather Station with Elixir and Nerveswas a joint effort between Bruce Tate, Frank Hunleth, and me.I have been writing software professionally for almost a decade and have been working primarily with Elixir since 2016. I currently maintain a few Elixir libraries onHexand also runStagira, a software consultancy company.Bruce Tateis a kayaker, programmer, and father of two from Chattanooga, Tennessee. He is the author of more than ten books and has been around Elixir from the beginning. He is the founder ofGroxio, a company that trains Elixir developers.Frank Hunlethis an embedded systems programmer, OSS maintainer, and Nerves core team member. When not in front of a computer, he loves running and spending time with his family.About the projectIn the Pragmatic Bookshelf book,Build a Weather Station with Elixir and Nerves, we take a project-based approach and guide the reader to create a Nerves-powered IoT weather station.For those unfamiliar with the Elixir ecosystem,Nervesis an IoT framework that allows you to build and deploy IoT applications on a wide array of embedded devices. At a high level, Nerves allows you to focus on building your project and takes care of a lot of the boilerplate associated with running Elixir on embedded devices.The goal of the book is to guide the reader through the process of building an end-to-end IoT solution for capturing, persisting, and visualizing weather data.Assembled weather station hooked up to development machine.One of the motivating factors for this book was to create a real-world project where readers could get hands-on experience with hardware without worrying too much about the nitty-gritty of soldering components together. Experimenting with hardware can often feel intimidating and confusing, but with Elixir and Nerves, we feel confident that even beginners get comfortable and productive quickly. As a result, in the book, we leverage a Raspberry Pi Zero W along with a few I2C enabled sensors to capture weather and environmental data. In all, the reader will capture and persist into TimescaleDB the current: altitude, atmospheric pressure, temperature, CO2 levels,TVOClevels, and the ambient light.Once the environmental data is captured on the Nerves device, it is published to a backend REST API and stored in TimescaleDB for later analytics/visualization. Luckily, TimescaleDB is an extension on top of PostgreSQL, allowing Elixir developers to use existing database tooling likeEctoto interface with time-series enabled tables.After the time-series weather data is stored in TimescaleDB, we walk the reader through how to visualize this data using the popular open-source visualization toolGrafana.Using Grafana for visualizing the weather was an easy choice given that Grafana natively supports TimescaleDB and is able to easily plot time-series data stored in TimescaleDB hypertables.✨Editor’s Note:Check outGrafana 101 video seriesandGrafana tutorialsto learn everything from building awesome, interactive visualizations to setting up custom alerts, sharing dashboards with teammates, and solving common issues.The diagram shows all of the various components of the weather station system and how they interact with one another.By the end of the book, readers have a fully-featured IoT application and API backend that can power a live Grafana dashboard in order to plot their TimescaleDB data published from their Nerves weather station.Screenshot of Grafana dashboard, showing various graphs for various weather data📖If you are interested in learning about how to build an end-to-end IoT weather monitoring solution, be sure tocheck out the book and the accompanying code. If you are interested in learning more about Nerves and Elixir,check out the Nerves documentation.Choosing (and using!) TimescaleDBFrom the onset of the book, we knew that we wanted to use a purpose-built time-series database to persist the weather station data. We wanted the project to be as realistic as possible and something that could possibly be expanded for use in the real world.With that goal in mind, TimescaleDB was an obvious choice given that PostgreSQL has become a ubiquitous database and it has great support in the Elixir community. In addition,leveraging TimescaleDB on top of PostgreSQL does not add a tremendous amount of overhead or complexity and allows new users to easily leverage the benefits of a time-series database without having to learn any new query languages or databases. Specifically, all it took for readers to start leveraging TimescaleDB was to run a single SQL command in their database migration:SELECT create_hypertable('weather_conditions', 'timestamp').""It’s this kind of pragmatism and ease of use that makes TimescaleDB a great time-series database for projects both small and large. ""-Alexander KoutmousAll in all, leveraging TimescaleDB as the time-series database for the project worked out great and allowed us to show readers how they can set up a production-ready IoT project in a relatively short amount of time.✨Editor’s Note:To start with TimescaleDB today,sign up for a free 30-day trialorinstall TimescaleDB on your own server.Getting started advice & resourcesAny time we had questions about the inner workings of TimescaleDB, how to set it up, or what the various configuration options are, we turned to the official TimescaleDB docs. Some of the articles that helped us get started included:•Using TimescaleDB via Docker• Understanding some ofthe fundamental TimescaleDB concepts• Getting an overview of some of theTimescaleDB best practicesWe’d like to thank Alex, Bruce, and Frank for sharing their story, as well as for writing a book that makes building full-stack IoT solutions accessible for complete beginners. We congratulate them and the entire Nerves community on their success, and we cannot wait to read the final version of their book that will be released in January 2022 🎊We’re always keen to feature new community projects and stories on our blog. If you have a story or project you’d like to share, reach out on Slack (@Lucie Šimečková), and we’ll go from there.Additionally, if you’re looking for more ways to get involved and show your expertise, check out theTimescaleHeroes program.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-build-a-weather-station-with-elixir-nerves-and-timescaledb/
2023-03-28T13:01:24.000Z,"Scaling PostgreSQL With Amazon S3: An Object Storage for Low-Cost, Infinite Database Scalability","Last November, we announced that we were building an integrated object storage layer to scale PostgreSQL further in a cost-effective way.Today, we’re happy to announce that data tiering is available in Early Access for all Timescale customers.By running a simple command on your hypertable (add_tiering_policy), you can automatically tier older data to a low-cost, infinite storage layer built on Amazon S3. Yet the data still remains fully queryable from within your database, and this tiering is transparent to your application.This storage layer is an integral part of your database. Your Timescale hypertables now seamlessly stretch across our regular and object storage, boosting performance for your most recent data and lowering storage costs for older data. There are no limitations to the data volume you can tier to object storage, and you’ll be charged only for what youstore—no extra charge per query or data read, and no hidden costs.We're also rolling out a special offer to celebrate this awesome achievement (we can’t wait for you to try this!).During the Early Access period, this feature will be entirely free for all Timescale customers, meaning that you won’t be charged for the volume of data tiered to the object store during that time. We’ll disclose our final pricing for tiered data in the following weeks, but don’t worry—you won’t lose what you saved:  the object storage will be priced roughly 10x cheaper than our regular storage. Stay tuned for details.This feature is a step forward to solving a pressing problem we keep hearing from developers:it is simply too expensive and difficult to scale PostgreSQL databases in AWS.Products like Amazon RDS for PostgreSQL and Amazon Aurora work great for small deployments, but once the project grows, databases in RDS and Aurora start getting prohibitively expensive. By allowing you to tier data to object storagewithout leaving Timescale, we avoid the need to remove data from your database in an attempt to lower costs or to avoid hard limits related to disk capacity.Instead,you now have an easy path for scaling sustainably and within budget without terabyte limitations, paying only for what you use.""We perform a lot of analysis on market data, and the sheer volume of data we need to store makes a normal disk-based database solution unfeasible (it's just too expensive).Timescale’s data tiering seamlessly allows us to access large volumes of data on S3. This is a great solution to store large volumes of historical data and perform post-analysis. Without this, we'd be forced to develop a solution in-house."" (Chief Technology Officer at a Proprietary Digital Assets Trading Company)With the object store integrated into your database, Timescale hypertables stretch effortlessly across multiple cloud-native storage layers. This architecture gives you a seamless querying experience—even when data is tiered, you can still query it within the database via standard SQL, just like in TimescaleDB and PostgreSQL. Everything “just works”: predicates, filters, JOINs, common table expressions (CTEs), windowing, andhyperfunctions.Data tiering is available for all TimescaleDB services in Timescale: you can start tiering data immediately (free trial included). Check out our documentation for instructions on enabling your first tiering policy, or see the video below.""RDS Pricing Is Too Steep: Why PostgreSQL Needs an S3 Object Store""Many of our customers come from PostgreSQL-compatible products, including Amazon RDS for PostgreSQL, Amazon Aurora, and Heroku. Even when these customers’ use cases vary across the board (from finance and IoT to energy), the story they tell us is consistently the same:At the start of their project, developers choose PostgreSQL due to its reliability, ease of use, and broad compatibility. As their first option, they often pick RDS PostgreSQL (the path of least resistance in AWS) or Heroku (which has an appealing price point for small deployments).Everything works great for a while, but as the project grows, so does the volume of data. That’s when the queries start slowing down.As time goes by and more data gets ingested, the problem becomes increasingly critical. The database starts holding the application hostage. The team works hard to optimize the database, but the performance improvements are only temporary.At this point, the teams running on RDS or Heroku sometimes choose to move their workloads to Amazon Aurora, which promises better performance than RDS. While the team hopes the move will solve their sluggish queries, another problem arises when they switch to Aurora: untenable, painfully high costs.Aurora’s billing proves to be unpredictable, with costs soaring much higher than anticipated (hello, I/O), and as the data volume increases, the problem only gets worse. There’s no ceiling for this ever-growing database bill, and growth quickly seems unsustainable.By moving to Timescale, teams experiencing these issues solve their performance and cost problems. With features like hypertables and continuous aggregates,Timescale delivers up to 350x faster queries and 44 % faster ingestion than RDS PostgreSQL. With such performance improvements, Timescale users can use a smaller compute footprint to accomplish similar workloads, leading to significant savings. Unlike Aurora,Timescale’s pricingis transparent and predictable. And with Timescale, customers can alreadyreduce their storage footprint by more than 95 % via columnar compression, saving money and further improving query performance.And yet, we knew we could do more by establishing a direct connection from Postgres to S3. We wanted to take advantage of the lower cost and reliability of Amazon S3 to offer a cheaper storage option to PostgreSQL users in AWS,adding one order of magnitude more savings to those already offered via our native columnar compression.Imagine we were engineers scaling a data-centric application—this is what we would build for our own use. We did it, so you didn’t have to.Infinite, Low-Cost Database Scalability for PostgreSQLAll data inserted into Timescale is initially written into our faster storage layer built on the latest-generation, IO-optimized EBS. Using faster disks for your most recent data will bring you top insert and query performance for your most recent values—a usage pattern that fits well for time series, events, and other analytics use cases. Once your data gets older (and mostly immutable), you can tier it to the object store automatically by defining a time-based policy. And it’sreallyeasy.# Create a tiering policy for data older than two weeks
SELECT add_tiering_policy ('metrics', INTERVAL '2 weeks');Hypertables now transparently stretch across two different storage layers, enabling fast performance for your most recent data and affordable, infinite storage for your older dataBy building this, we’re providing a low-cost alternative for scaling your PostgreSQL databases in AWS. This is our special offer during the Early Access period:You won’t be charged for the data in object storage. For the next 1-2 months, data tiering will be free for all users.You won’t stop saving once the free period is over. The object storage will be roughly 10x cheaper than our regular storage.There are no limitations to the volume of data you can tier to object storage, and you will be charged only per gigabyte—no extra charges per query or other hidden costs.A Postgres Object Store Built on Amazon S3: Much More Than an External ArchiveThis object store is much more than an external bucket to archive your data: it’s an integral part of your database. When tiering data, your database will remain fully aware of all the semantics and metadata. You can keep querying as usual with standard SQL.With Timescale’s data tiering, all data tiered to S3 is in a compressed columnar format (specifically,Apache Parquet).  When data is tiered to S3 based on its age, chunks stored in Timescale’s native internal database format (typically itself in ournative columnar compressionstored in a Postgres-compatible data table) are asynchronously converted to Parquet format and stored in S3.  These tables remain fully accessible throughout the tiering process, and various mechanisms ensure that they are durably stored to S3 before transactionally removed from standard storage.When you run your SQL query, it will pull data from the disk storage, object storage, or both as required. And to avoid processing chunks falling outside the query’s time window, we perform “chunk exclusion” to our query those chunks minimally required to satisfy a query.Further, the database is selective in what data is read from S3 to improve query performance; it stores various forms of metadata to build a  “map” of row groups and columnar offsets within the S3 object.  If your query only touches a range of rows and few of the columns, only that subset of the data is read from the tiered S3 object. The result? Less data to fetch and thus faster queries.And when we say transparent, we mean transparent. Timescale supports arbitrarily complex queries across its standard and tiered data, including complex predicates, JOINs, CTEs, windowing, hyperfunctions, and more.In the example query below (with an EXPLAIN clause), you can see how the query plan includes a `Foreign Scan` when the database is accessing data from S3. In this example, three chunks were read from standard storage, and five chunks were read from object storage.EXPLAIN 
SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;


QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
    -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text)How to Get StartedStart slashing your storage bill today!Data tiering is available in Early Access for all Timescale customers. To activate it, simply navigate to your service’s Overview page and press “Enable data tiering.” For detailed instructions, see our documentation or watch our demo video below:Haven’t tried Timescale yet?Start a Timescale trial and get full access to the platform for 30 days, no credit card required.Data tiering is also available for trial services: start experimenting now!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/scaling-postgresql-with-amazon-s3-an-object-storage-for-low-cost-infinite-database-scalability/
2022-02-10T14:12:05.000Z,Listening to the Developer Community Yields Benefits: Introducing Timescale’s Community Forum,"Developer products like TimescaleDB are made more useful and powerful when listening and acting on the needs of the developer community. Timescale believes that developers drive the success of companies like ours through product adoption and advocacy. Successful developer communities might be facilitated by companies, but their members drive them. And the health and vibrancy of a community are a consequence of developer-to-developer conversations and collaboration.From the outset, we’ve understood the importance and power of the developer community, and adopted Slack as the main channel for community discussion and influence. The immediate feedback loop worked well. Timescale met a lot of great people along the way and has some awesome community contributors helping less experienced users with the phenomenal features of TimescaleDB. (Thank you!)The feedback you provide via Slack and GitHub issues often drives our product development. For example in 2021, we delivered the ability toinsert into compressed chunks, and experimental features to provide for variable sized time buckets and timezone support for time buckets viatime_bucket_ngin response to requests from our community.However, as anyone who joined ourcommunity day session with Chris Engelbert and Ben Wilsonwill know – and if you’ve been around TimescaleDB’s Slack for a while – there are several discussions that recur frequently. We’ve also received community calls for the chance to create more structured and longer-lasting discussion threads. Not everyone enjoys corresponding in chat channels, and for our global community, conversational short-form chat is not always the best medium. Lots of projects offer both a chat channel and something more along the lines of a traditional forum, and we’ve had several requests from users for us to launch a forum for Timescale.Screenshot of Timescale Community ForumTime TravelersAs part of our work to make our community a great place to exchange knowledge about these database application technologies, we’re revamping our community recognition program. We want to recognize contributors big and small because all contributions to the Timescale community can have a positive impact. Our talented designer Julia Nasser had made some awesome drawings of Eon for ourNFT Collection of Time Travel Tigers, and that sparked an idea for the theme of our new program–Timescale Time Travelers.The Time Traveler program enables everyone in the community to travel with us as we continue to learn, build, and develop Timescale… Asking questions, giving answers, writing posts, contributing code, editing docs… there’s a multitude of ways that valuable contributions can be made to the community. While being helpful has its own rewards—and with the new public forum, you can share evidence of your community spirit and knowledge—we’re also creating some unique SWAG to distribute to our most active community members. More about this soon!It takes a village…Looking back to my owncommunity day discussion, as Timescale’s community manager, I want to make sure that all voices have the chance to be heard and that everyone has a place where they can get the information they need in a format that works for them. The new forum and the Time Traveler program move us along the path of helping you as developers get the most out of your database applications.If Timescale’s a place where you can build your knowledge, discuss things that matter to you, and meet other developers that share your interests, then we are doing our job.For those who already use our Slack community, I’ve posted a link to the forum in the#announcementschannel… using that link, you’ll get a special welcome, and a thank you 🎁 from us as a Timescale OG! Also note we will continue using Slack and make sure conversations of note are highlighted in both forums.For those who are not currently using Slack, you’rewelcome to joinus there, but you might also enjoy taking part in ournew community forum. Sign up now!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/listening-to-the-developer-community-yields-benefits-introducing-timescales-community-forum/
2022-03-28T13:06:01.000Z,High Availability for Your Production Environments: Introducing Database Replication in Timescale,"Earlier this year, we kicked off the Year of the Tiger 🐯🦄 by announcing our$110M Series C fundingto build the future of data for developers worldwide. This new funding helps us accelerate the delivery of features that help our customers build best-in-class data-driven applications. This week, we’re excited to continue this momentum with #AlwaysBeLaunching (#ABL): MOAR Edition—a week full of exciting new features for Timescale, bringing you MOAR features that makeTimescaleeven MOAR worry-free, scalable, and flexible!To start, we’re releasing early access to a highly requested feature:database replication in Timescale.And throughout the rest of this #ABL Cloud Week, we’ll release features that give you MOAR collaboration, visibility, performance, and regions in Timescale.One of our guiding principles as a company is that boring is awesome and thatyour database should be boring: we believe that you should be able to focus on your applications rather than on the infrastructure on which they’re running. Replicas is a feature commonly available in most databases, and today we’re surfacing that functionality in an easy-to-use experience within Timescale. As with all things Timescale, we build on top of the proven functionality available in PostgreSQL, the foundation of TimescaleDB.Database replication in Timescale is as easy as pressing a button. By enabling a replica, you will increase the availability of your data, assuring that you will only experience a few seconds of downtime if your database fails—but that’s not all. In Timescale, enabling replicas can also improve performance, as you can easily direct your heavy read queries to the replica, which frees up resources in your main database for higher ingest rates, more advanced continuous aggregates, or additional read queries.The entire database replication process (including creating a replica, checking out its status, retrieving its URL, and deleting your replica) can be done easily and transparently from the Console UI. And if your database happens to fail – for example, because the underlying AWS server instance becomes unavailable – all the recovery processes are done automatically. Timescale will shift responsibilities from the former failed primary to the replica, which will then be elevated to start accepting write operations as a new primary without any action needed from you. Your operations will just keep on running while Timescale does this heavy lifting under the hood.Read on to learn more about replicas in Timescale, and how you can use database replication to increase data availability and liberate load in your Timescale database.If you’re new to Timescale,try it for free(100% free for 30 days, no credit card required). Once you’re up and running, join our community of 8,000+ developers passionate about TimescaleDB, PostgreSQL, time-series data, and all its applications! You can find us in theTimescale Community Forumandour Community Slack.A huge “thank you” to the teams of engineers and designers that made all the features we’re releasing during the #AlwaysBeLaunching a reality! 🙏Replicas in TimescaleYour database should be worry-free: you shouldn't have to think about it, especially if there’s a problem. As any database operator knows, failures occur. But a modern database platform should automatically recover and re-establish service, as soon as issues arise. Our commitment is to build a dependable, highly-available database that you can count on. Boring is awesome when the platform prevents you from getting paged at 3 am.Many architectural aspects of Timescale are intentionally aligned with this high availability goal. For example, Timescale’s decoupled compute and storage is not only great for price optimization but also for high availability. In the face of failures, Timescale automatically spins up a new compute node and reconnects it to the existing decoupled database storage, which itself is independently replicated for high availability and durability. Indeed, even without a replica enabled, this cloud-native architecture can provide a full recovery for many types of failures within 30-60 seconds, with more severe physical server failures often taking no more than several minutes of downtime to recover your database.Further, incremental backups are taken continuously on Timescale for all your services (and stored separately across multiple cloud availability zones), allowing your database to be restored to any point in time from the past week or more. And Timescalecontinuously smoke tests and validates all backupsto ensure they are ready to go at a moment’s notice.However, many customers run even their most critical services on Timescale, where they need almost zero downtime—even in the case of unexpected and severe hardware failures. Indeed, Timescale powers many customer-facing applications, where downtime comes with important consequences for the business: application dashboards stop responding, assembly lines are no longer monitored, IoT sensors can no longer push measurements, critical business data can be lost, and more. Database replication provides that extra layer of availability (and assurance) these customers need.Apart from decreasing downtime, Timescale replicas have another advantage: they can also help you ease the load from your primary database. If you are operating a service subjected to heavy read analytical queries (e.g., if you’re using tools like Tableau or populating complex Grafana dashboards), you can send such read queries to the replica instead of to your primary database, liberating its capacity for writes and improving performance. This makes your replica useful even in the absence of failure.And leveraging this functionality is as easy as using a separate database connection string that Timescale makes available: one service URL for your (current) primary, and the platform transparently re-assigns this connection string to a replica if that replica takes over and a second service URL that maps to your read-only replica.If you are already a Timescale user, you can immediately set up a database replica in your new and existing services. Enabling your first replica is as simple as this:Select the service you’d like to replicate.Under “Operations,” select “Replication” on the left menu.To enable your replica, click on “Add a replica.”That’s it! 🔥If a service has a replica enabled, it will show under Operations -> Replication. (Take into account that the replica won’t show in your Services screen as aseparate database service,as it is not an independent service.)You can enable a replica in Operations -> Replication. Once created, your replica status will show on this page.If you want to direct some of your read queries to the replica, go to your service's “Overview” page. Under “Connection info,” you will see a drop-down menu allowing you to choose between “Primary” and “Replica.” To connect to your replica, you can simply select “Replica” and use the corresponding service URL.To connect to your replica, select Replica on your Overview page and use the corresponding service URL.We’re releasing database replication with an early access label, meaning that this feature is still in active development. We will add new functionality to replicas in the very near future: at the end of this post, you will find a detailed list of everything we’re actively working on regarding replicas.You’ll hear from us again soon!How Replicas WorkReplicas are duplicates of your main database, which in this context is called “primary.” When you enable a replica, it will stay up-to-date as new data is added, updated, or deleted from your primary database. This is a major difference between a replica and a fork,which Timescale also supports: a fork is a snapshot of your database in a particular moment in time, but once created, it is independent of your primary. After forking, the data in your forked service won’t reflect the changes in the primary.PostgreSQL (and thus TimescaleDB) offersseveral methods for replication. However, setting up replication for a self-hosted database is a difficult task that includes many steps—from choosing which options suit you best, to actually spinning up a new server and adjusting configuration files, to tweaking configurations, and to building a full infrastructure that monitors the health of your primary and automatically failing-over to a replica when needed.  All the while avoiding “split-brain” scenarios in which two separate services both believe they are primaries, leading to data inconsistency issues.Timescale automates the process for you: we do the hard work so you don’t have to.As we’ve seen in the previous section, adding a replica to your service is extremely simple. But as a more technical deep dive for the interested reader, the following paragraphs cover three design choices we’ve taken for replicas: (i) the choice of asynchronous commits, (ii) their ability to act as hot standbys, and (iii) their use of streaming replication.Timescale replicas are asynchronousThe primary database will commit a transaction as soon as they are applied to its local database, at which point it responds to a requesting client with success.  In particular, it does not wait until the transaction is replicated and remotely committed by the replica (as would be the case with synchronous replicas). Instead, the transaction is asynchronously replicated to the replica by the primary shipping its Write-Ahead Log (WAL) files, hence the “quasi-real-time” synchronicity between primary and replica.We chose this design pattern for two important reasons. The first one, perhaps surprisingly, is high availability: with a single synchronous replica, the database service would stop accepting new writes if the replica fails, even if the primary remains available.  The second is performance, as database writes are both lower latency (no round trip to the replica before responding) and can achieve higher throughput.  And that’s important for time-series use cases, where ingest rates are often quite high and can be bursty as well.In the future, when we add support for multiple replicas, we plan to introduce the ability to configure quorum synchronous replication, where a transaction is committed once written to at least some replicas, but not necessarily all. This addresses one tradeoff with asynchronous replication, where a primary failure may lead to the loss of a few of the latest transactions that have yet to be streamed to any replicas.Timescale replicas act like hot standbysAwarm standbymeans that the replica (standby) is ready to take over operations as soon as the primary fails, as opposed to a “cold standby” which might take a while to restore before it can begin processing requests.  This is closely related to the high availability mentioned earlier. The process of the standby/replica becoming the primary is called failover, which is covered more below. Since Timescale replicas are also read replicas – i.e., they can also be used for read-only queries – they are consideredhot standbysinstead of justwarmstandbys.As we will talk about in later sections, allowing you to read from your replicas gives you the option to direct some of your read-only workloads to your replica, freeing capacity in your primary. This means that in Timescale, you will not only have a replica ready to take over at any moment if the primary happens to fail, but you can also get value from it beyond availability, even if there’s no failure.Timescale replicas use streaming replicationStreaming replication helps ensure there is little chance of data loss during a failover event.  Streaming replication refers to how the database’s Write-Ahead Log – which records all transactions on the primary – is shipped from the primary to the replica. One common approach for shipping this WAL is aptly-named “log shipping.” Typically, log shipping is performed on a file-by-file basis, i.e., one WAL segment of 16MB at a time. So, these files aren’t shipped until they reach 16MB or hit a timeout.The implication of log shipping, however, is that if a failure occurs, any unshipped WAL is lost. Instead of file-based log shipping, Timescale uses streaming replication to minimize potential loss. This means that individual records in the WAL are streamed to the replica as soon as they are written by the primary rather than waiting to ship as an entire segment. This method minimizes the potential data loss to the gap between a transaction committing and the corresponding WAL generation.Enabling Replicas for High AvailabilityEven without a replica enabled, Timescale has a range ofautomated backup and restore mechanismsthat protect your data in case of failure. For example, the most common type of failure in a managed database service is a compute node failure; in Timescale, it often takes only tens of seconds to recover from such a failure, as we are able to spin up a new compute node and reattach your storage to it. In the much rarer case, in which a failure affects your (replicated) storage, Timescale automatically restores your data from backup, at a rate of roughly 10 GB per minute.For some use cases, the potential level of downtime associated with this recovery process is completely acceptable; for those customer-facing applications that require minimal downtime, replicas will provide the extra layer of availability they need.The recovery process through replicas is summarized in the figures below. In a normal operating state, the application is connected to the primary and optionally to its replica to scale read queries.  Timescale manages these connections through load balancers, defining the role for each node automatically.In a normal operating state, the application is connected to the primary and optionally to its replica. The load balancer handles the connection and defines the role for each node.The next figure illustrates a failover scenario. If the primary database fails, the platform automatically updates the roles, “promoting” the replica to the primary role, with the primary load balancer redirecting traffic to the new primary. When the failed node either recovers or a new node is spun up, it assumes the replica role. The promoted node remains the primary, streaming WAL to its new replica.When the primary database fails, the platform updates the roles, “promoting” the replica to the primary role, with the primary load balancer redirecting traffic to the new primary. In the meantime, the system begins the recovery of the failed node.When the failed node either recovers or a new node is spun up, it assumes the replica role. The promoted node remains the primary, streaming WAL to its new replica.When the failed node recovers or a new node is created, it assumes the replica role. The previously promoted node remains the primary, streaming WAL to its replica.On top of increasing the availability of your database in case of failure, replicas also essentially eliminate the downtime associated with upgrades, including database, image, or node maintenance upgrades. Without a replica, these upgrades usually imply 30 to 60 seconds of downtime. With a replica, this is reduced to about a second (just the time to failover). In this case, when the upgrade process starts, your system will switch over to the replica, which now becomes the primary. Once the upgrade is completed in the now-replica-formerly-primary, the system switches back so it can subsequently upgrade the other node. (And on occasions the replica is upgraded first, in which case only one failover will be necessary.)Read Replicas: Enabling Replicas for Load ReductionTimescale's replicas act as “hot standbys” and thus also double as read replicas: when replicas are enabled, your read queries can be sent to the replica instead of the primary.The main advantages of read replicas are related to scalability. By allowing your replica to handle all of the read load, your primary instance only has to handle writes or other maintenance tasks that generate writes, such as TimescaleDB’s continuous aggregates. Using read replicas would result in higher throughput for writes and faster execution times on analytical reads, plus a less strained primary instance.For example, read replicas can be particularly useful if you have many Grafana dashboards connecting to your service. Since visualizations don’t need perfectly real-time data – that is, using data that’s a few seconds old is often more than acceptable – the replica can be used to power these dashboards without consuming resources on the primary. Plus, with this setup, data analysts can work with up-to-date production data without worrying about accidentally impacting the database operations, such as with more ad-hoc data science queries.Another benefit of using read replicas is to limit the number of applications with write access to your data. Since the entire replica database is read-only, any connection, even those with roles that would have write privileges in the primary, cannot write data to the replica. This can serve to easily isolate applications that should have read/write access from those that only need read access, which is always a good security practice.  Database roles should certainly also be used to ensure “least privilege,” but a bit of redundancy and “defense in depth” doesn’t hurt.Coming SoonToday, we’re releasing database replication under an “early access” label, meaning that this feature is still in active development. We’ll be continuing to develop capabilities around database replication in Timescale, including:Multiple replicas per database serviceGreater flexibility around synchronous vs. asynchronous replicasReplicas in different AWS regionsReplicas in multi-nodedatabaseservicesSo keep an eye out – MOAR replication options coming soon!Get StartedTimescale’s new database replication provides you with increased high availability and fault tolerance for your important database services. In addition, it allows you to scale your read workloads and better isolate your primary database for writes. Check out our documentation for more information on how to use this database replication in Timescale.Replicas are immediately available for Timescale users. If you want to try Timescale,you can create a free account to get started—it’s 100% free for 30 days, without a credit card required. And if you have any questions, you can find us in ourCommunity Slackand also in theCommunity Forum.And if this sounds like the type of technical challenges you enjoy working on:We’re hiring. Fully remote and globally distributed.So let’s kick off the first Cloud Week 2022 with MOAR availability. And stay tuned – many more exciting Timescale capabilities to come!🐯🚀Eon, our Timescale mascot, roars “MOOARRRR” in honor of #AlwaysBeLaunching!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/high-availability-for-your-production-environments-introducing-database-replication-in-timescale-cloud/
2023-11-08T13:00:00.000Z,Create Timescale Services With the Timescale Terraform Provider,"📣The Timescale Terraform provider hits 1.0 and is available for all production use. Learn how to use it.Why Terraform?Terraform is a must-have tool in any software developer’s toolbox. It allows for declarative configuration of cloud (and sometimes on-prem) resources. If used correctly, your cloud infrastructure is a true one-to-one representation of your configuration files and vice versa.One of the biggest benefits of Infrastructure as Code (IaC) is increasing visibility into what your deployments really look like, which in turn helps with security, auditability, and accountability. My personal favorite benefit is how it increases the speed of deployment. Say goodbye to click-ops, because the speed at which Terraform allows you to deploy, reuse, and recycle configurations is supersonic.Before joining Timescale as a developer advocate, I was a DevOps engineer at a British tech startup in the renewable energy sector. A significant part of my time here was spent deploying our infrastructure and software stack across fresh cloud accounts.The infrastructure was entirely made up of Terraform modules that allowed me to mix and match required AWS services akin toBuild-A-Bear. At most, my manual labor was limited to configuring the subnet CIDRs, modifying some DNS entries, and creating some SSH keys.Everything else was done automatically, from deploying EC2 instances and Kubernetes clusters to databases and their tables. Thanks to Terraform’s idempotent nature, if something went wrong, all I had to do was re-run theTerraform deploycommand, and things would (usually) fix themselves.We were able to move at a breakneck speed by leveraging Terraform, far surpassing the tedious process of manually deploying and configuring resources in the AWS Console. Needless to say, Terraform is an invaluable asset, having saved me countless weeks, if not months, of boring and repetitive work.So, when Timescale announced internally they were working on their own Terraform provider, I was on cloud nine.Now, several months later, the Timescale Terraform provider has reached 1.0 and has officially been released to General Availability, and I couldn’t be happier with it!✨Editor's Note:Check out this blog post to learn more aboutTerraform providers.To celebrate this launch, I thought it would only be appropriate to write a blog post about what a Terraform provider is and how you can use the new Timescale provider to deploy and destroy Timescale services.Configuring the Terraform providerTo use the Timescale Terraform provider, you must declare it in arequired_providersblock nested inside aterraformblock. Set the source totimescale/timescaleas that corresponds to the official provider in the Terraform Registry.While not strictly required, you can configure Terraform to use a specific version of the Timescale provider. To use the latest version, use the~>operator or omit the version argument altogether.terraform {
  required_providers {
    timescale = {
      source = ""timescale/timescale""
      version = ""1.1.0""
    }
  }
}After declaring the Timescale provider, you need to configure it with theProjectIDand credentials.provider ""timescale"" {
  project_id = var.ts_project_id
  access_key = var.ts_access_key
  secret_key = var.ts_secret_key
}While safely managing credentials in Terraform is a topic for another blog post, I like to keep it simple and create three variables for each parameter in a separate file. Though Ihighlyrecommend using a safer non-plain text solution like Hashicorp Vault, Terraform environment variables, or AWS KMS.variable ""ts_project_id"" {
  default    = ""your-project-id""
}

variable ""ts_access_key"" {
  default    = ""your-access-key""
}

variable ""ts_secret_key"" {
  default    = ""your-secret-key""
}To retrieve theProjectID, go to the Services tab in Timescale and click on the kebab menu in the top right corner.To create credentials, go to theProject Settingstab and clickCreate credentials. A window will pop up with your public and private keys. Make sure you copy both the public and private keys to a secure location (like a password manager), as this is the only time you will be able to retrieve them.It’s also paramount to never share the private key with anyone you don’t 100 percent trust, as it would allow them to create and delete Timescale services at will until the credentials have been deleted.You’ve successfully configured the Timescale provider; you can now create your first Timescale service using Terraform!Creating a serviceIf you read the documentation, you’ll find that configuring a Timescale service through Terraform is very similar to creating a service through the advanced configuration menu in the Timescale UI.First and foremost, you can give your Timescale service a name. Do note that the Terraform resource name and `name` parameters don’t need to be the same.Next up is the CPU and memory configuration. Just like in the Timescale UI, only a handful of CPU and memory combinations are supported. Please consult theprovider documentationfor more information.In this case, we have configured our Timescale instance with 1 CPU and 4 GB of memory. Timescale automatically allocates and charges only for the storage you use, therefore you don’t pre-provision a specific storage size. Please consult theprovider documentationfor more information.For the region, I have chosenus-east-1, but you can choose one of oursupported regions.Lastly, I added aprevent_destroylifecycle. This is a feature native to Terraform that prevents you from accidentally destroying your service. If a Timescale service is destroyed, the data it stores gets destroyed too, and because data loss is something we want to avoid at all costs, I think this is an absolute must.If you do eventually want to delete your Timescale service and the corresponding data, you can remove or comment out the lifecycle.resource ""timescale_service"" ""my-resource"" {
  name        = ""my-resource""
  milli_cpu   = 1000
  memory_gb   = 4
  region_code = ""us-east-1""

  lifecycle {
    prevent_destroy = true
  }
}And that’s it! We are now ready to deploy our service! Execute the following commands in order. Enteryeswhen Terraform prompts you to confirm the actions.terraform init
terraform plan
terraform apply

Terraform will perform the following actions:

  # timescale_service.my-resource will be created
  + resource ""timescale_service"" ""my-resource"" {
      + enable_ha_replica          = false
      + hostname                   = (known after apply)
      + id                         = (known after apply)
      + memory_gb                  = 4
      + milli_cpu                  = 1000
      + name                       = ""my-resource""
      + password                   = (sensitive value)
      + port                       = (known after apply)
      + region_code                = ""us-east-1""
      + username                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

timescale_service.my-resource: Creating...
timescale_service.my-resource: Creation complete after 38s [id=wcdokcbcd4]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.Retrieving the service passwordGreat! You’ve successfully deployed a Timescale service using Terraform. But how do you access your new service? Your first intuition might be to go to the Timescale UI and copy the connection string, but as you’ll soon notice, that string doesn’t include the password. Theonlyway you can get the password is by using outputs!output ""service_hostname"" {
  value = timescale_service.my-resource.hostname
}

output ""service_port"" {
  value = timescale_service.my-resource.port
}

output ""service_username"" {
  value = timescale_service.my-resource.username
}

output ""service_password"" {
  value     = timescale_service.my-resource.password
  sensitive = true
}After adding the outputs, re-apply your configuration using the apply command and use the output command to retrieve the outputs you just created.terraform apply
terraform outputAs you can see, Terraform will gladly show you the hostname, port, and username, but it keeps the password hidden. This is because your service's password is a sensitive value, and it would be insecure to output it to the terminal on everyterraform applybecause you never know where your logs will end up! To retrieve the password, we can use-jsonflag to make Terraform print the outputs in JSON format.terraform output -json
{
...
  ""service_password"": {
    ""sensitive"": true,
    ""type"": ""string"",
    ""value"": ""vswm7wg2ls3cqpvd""
  }
...
}The wall of text it prints out into the terminal is less than ideal, so with another tool calledjqwe can filter out just the password.terraform output -json | jq -r "".service_password.value""If you know Terraform well enough, you can even synthesize the entire connection string as an output. This could be super useful when deploying other resources using Terraform that connect to your Timescale service.output ""service_url"" {
  value = format(""postgres://tsdbadmin:%s@%s:%s/tsdb?sslmode=require"",
    timescale_service.my-resource.password,
    timescale_service.my-resource.hostname,
  timescale_service.my-resource.port)
  sensitive = true
}Having the connection string pre-synthesized also allows you to connect to your Timescale service using psql in a single command without even needing to print the password to the terminal!psql $(terraform output -json | jq -r "".service_url.value"")Destroying a serviceWhen you’re all done with your Timescale service, you can shut it down using theterraform destroycommand. But not before you remove or comment out theprevent_destroylifecycle rule. Keep in mind that destroying your Timescale service will permanently delete your data!terraform destroyWhen the destroy command completes, the services you created will be gone.Start Using the Timescale Terraform ProviderIf, like myself, you’re using Timescale as your cloud database and are a fan of the convenience and speed an open-source IaC tool like Terraform can bring, I know you’ll be as excited as I am about the Timescale Terraform provider.Most developers are looking for tools that will make their work simpler, faster, and less tedious. However, they struggle to find them, making the dream of speedy deployments and time spent building instead of configuring a near impossibility.At Timescale, we want to make it possible for software engineers to focus on their applications, not their databases. With the recent addition of the Timescale Terraform provider, managing your services with ease is now a reality.Try it yourself: sign up for a free 30-day trial(no credit card required).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/create-timescale-services-with-terraform-provider/
2023-10-30T12:24:57.000Z,Refining a Mature Cloud Platform: Cloud Week at Timescale,"When we first launched TimescaleDB seven years ago, our message was straightforward:  “Something as old as PostgreSQL can seem boring. But sometimes boring is awesome, especially when it’s your database. TimescaleDB is designed to just work, not wake you up at 3 a.m.”Timescale scales PostgreSQL’s performance for demanding applications. Tens of thousands of businesses power their products with TimescaleDB and build real-time in-app dashboards and analytics on top of their sensor data, application data, financial data, and more.Timescale allows developers to stay with PostgreSQL even as their workloads scale, rather than risking a replatforming to a more unconventional and unfamiliar database engine.But many of today’s developers are not just looking for great database software but a fully managed, mature cloud database platform. Cloud databases that are easy to start, easy to use, and easy to scale, even while offering the performance, reliability, availability, security, operational control, and cost-effectiveness that businesses need, whether a new startup or a large enterprise. And that intangible awesomeness of running smoothly each and every 3 a.m. throughout the year.That’s how our cloud has evolved since we started building it in 2019. A mature cloud platform that is both innovative and dependable, and one worthy of the trust that our users place in us every day to power their core businesses. Ready for the builders. Ready for the enterprise.Guess what? It’sCloud Weekhere at Timescale.What’s New This Week?Today kicks off the first of three consecutive launch weeks here at Timescale, announcing a myriad of capabilities—both big and small—we’ve been developing and maturing over the past months (and in a few cases, years).The first one, Cloud Week, focuses particularly on the needs of production users and businesses. We’ll be describing these features in more detail throughout the week on their actual launch dates, but at a high level, this week’s launches will include:A newEnterprise Tierto serve the unique needs of large enterprises as they use Timescale, including additional security, reliability, and compliance features for these customers.An amazingInsightsproduct(to General Availability) that powers the deep observability we now offer on Timescale, at a granularity not offered by other platforms.Flexible, fully managedConnection Pooling(to General Availability) to serve high-scale, bursty, and serverless workloads.One-clickPoint-in-time Recovery and Database Forkingto more easily support development branching, testing, and other operational workflows.A newLive Migrationstoolto enable zero downtime migrations of PostgreSQL and self-managed TimescaleDB databases to our cloud.And maybe another small surprise or two.Built for ProductionBut before we properly introduce any of our new features—stay tuned for upcoming blog posts—a quick summary of a number of benefits that Timescale’s mature cloud platform offers for organizations looking to run their PostgreSQL workloads in the cloud. It’s been a multi-year journey to get here, but the team is proud of its robust capabilities today:Best-in-class performance:Timescale is Postgres, but faster. It’s built on the foundation of 100 % open-source PostgreSQL, but adds new capabilities through our embedded TimescaleDB database extension that yields lower query latency and higher throughput. And a cloud purpose-built with TimescaleDB and performance in mind. But just see the open-source benchmarks, with3.3x to over 300x faster queries than Amazon RDS for PostgreSQL.Scalability made easy:With Timescale, you can easily scale your databases in multiple dimensions, as compute and storage scale independently. Compute scales either vertically (to large compute sizes) or horizontally (by adding read replicas), and Timescale’s storage automatically grows or shrinks with your usage, from 10 GB to hundreds of TBs of data. This flexibility allows your database to easily adapt to changing workloads.High availability and reliability:Timescale provides automated backup and recovery through continuous incremental backup,regular database and disk snapshots, and point-in-time recovery and branching. It offers multi-AZ (availability zone) deployments forhigh availability, reducing the risk of downtime, and rapid recovery for all services by fast database restart and remote disk remount given its disaggregated architecture. Further, it employs unique memory guard protections to avoid PostgreSQL out-of-memory crashes and decouples its control and data planes for greater resilience with availability guarantees backed by commercial service level agreements (SLAs).Automated upgrades and software patching:Timescale takes care of applying database updates and security patches, reducing the risk of security vulnerabilities and ensuring that your database is up-to-date. TimescaleDB and minor PostgreSQL upgrades are applied with zero downtime during configurable maintenance windows, and PostgreSQL major version upgrades use a forking workflow and disk snapshots to minimize downtime and risk. By reducing operational overhead, your developers and ops team can focus on more strategic and value-added activities.Robust security and compliance:Timescale treats security seriously, from formal compliance (SOC 2 Type 2 and GDPR) to platform features. Data encryption at rest (both disk and backups) and in transit, database role-based access control, platform SSO/SAML and multi-factor authentication, and secure networking, including VPC peering. Timescale employs layered database “escalation” protections, secure SDLC practices, automated vulnerability scanning and code analysis, third-party pen-testing, and other measures to protect your data and ensure compliance with industry regulations.Deep observability:Timescale provides deep operational visibility into your databases, both to help understand performance, uncover regressions, and optimize performance. This isn’t just your standard in-database statement logs, but deep system-level capture and correlation at unprecedented scale and fidelity. Further, metrics and logs can be continuously exported toAWS CloudWatchorDatadogfor single-pane monitoring and alerting.Transparent, cost-effective pricing:Timescale offers simple, easy-to-understandconsumption-based pricingbased on compute and storage, with no hidden fees like data transfer costs, data read/scan costs, or others. Timescale is cost-effective compared to both managed databases like AWS RDS for PostgreSQL, and even self-managed PostgreSQL or TimescaleDB deployments. Cost savings emerge from both core database technologies (columnar compression, data tiering, resource efficiency), usage-based storage pricing that avoids wasteful disk allocation, and built-in operational features that reduce the need for dedicated database administrators.Top-notch support and services:Timescale employs an amazing team of PostgreSQL experts, staffed in a 24x7 follow-the-sun model across APAC, EMEA, and NASA, who provide full consultative support, including Production Support with Severity-1 responsiveness. Architectural reviews, data modeling and query optimization assistance, feature testing, and migration support all ensure customer success, with Customer Satisfaction (CSAT) scores regularly above 98 %.Global reach:Finally, you can deploy Timescale in many AWS regions worldwide, enabling you to reach a global audience while ensuring low-latency, secure access to your data.We hope this list piques your interest in what’s to come. So sit back this week, pop open your browser console and a handypsqlshell, and enjoy some new capabilities from Timescale.If you’re still not using Timescale,you can create your first service in just two minutes.And if you’d like personalized advice for your use case,reach out to us. We’d love to hear from you!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/refining-a-mature-cloud-platform-cloud-week-at-timescale/
2023-02-07T14:56:00.000Z,Important News About Promscale,"Dear community,We regret to inform you that we’ve made the difficult decision to discontinue Promscale.Promscale was created to help the community use PostgreSQL/TimescaleDB as a fully compliant remote storage for Prometheus. Later on, we added OpenTelemetry and Jaeger tracing support.With its full SQL support, it provided unified storage for traces and metrics with deep analysis and correlation capabilities. We also built tobs to simplify the deployment of a complete observability stack with one command.Promscale has been a labor of love for our team at Timescale for over two years, and we’re grateful for the support and feedback from you, our community of users. However, after reevaluating our company priorities, we’ve decided to redirect our resources to other product areas.We understand this news may be disappointing, so please see the FAQ for answers to common questions, including alternative solutions.Thanks again for your support.FAQsWhy is Promscale discontinued?We’re refocusing all our efforts on the core of what we do best, creating a great developer experience for time-series application developers.What happens next?The following GitHub repositories and documentation will remain publicly available but will no longer be evolved nor maintained:https://github.com/timescale/promscalehttps://github.com/timescale/promscale_extensionhttps://github.com/timescale/tobshttps://github.com/timescale/helm-charts/tree/main/charts/promscalehttps://promscale-legacy-docs.timescale.com/We’ll continue to provide support for critical issues to existing customers of Promscale with Timescale or MST until April 30th, 2023, at which point support will end.Can I keep using Promscale with TimescaleDB?You can continue to use Promscale with TimescaleDB, although we don’t recommend it since Promscale will no longer be supported or maintained.The GitHub repositories and release artifacts will remain publicly available.We can’t ensure that Promscale will continue to work with newer versions of TimescaleDB and PostgreSQL. If you want to make modifications, we encourage you to fork the repositories.Can I keep using the open-source version of Promscale with Timescale/ MST?You can until April 30th. Before April 30th, you should transition to a different solution since we can’t ensure that Promscale will continue working, and we plan to remove the unsupported Promscale extension from our servers. \Who do I contact if I am a customer of Timescale/ MST and I need critical support?Contact the support team at[email protected]orhttps://www.timescale.com/support.What Promscale alternatives do you recommend?There are different options depending on your use case.If you use Promscale as remote storage for Prometheus, there are a number of other options you can consider. There is a list provided in thePrometheus documentation.If you are using Promscale as a storage system for your OpenTelemetry or Jaeger traces,there are a number of options you can consider.This is a list of products that have native support for OpenTelemetry. You can use the OpenTelemetry Collector to convert your OpenTelemetry traces to Jager traces.If you have a specific need, you can ask in the #promscale channel inTimescale's Community Slack, and we’ll try to help you.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/important-news-about-promscale/
2022-06-21T12:58:38.000Z,How We Made Data Aggregation Better and Faster on PostgreSQL With TimescaleDB 2.7,"It’s time for another #AlwaysBeLaunching week! 🥳🚀✨ In our #AlwaysBeLaunching initiatives, we challenge ourselves to bring you an array of new features and content. Today, we are introducing TimescaleDB 2.7 and the performance boost it brings for aggregate queries. 🔥 Expect more news this week about further performance improvements, developer productivity, SQL, and more. Make sure you follow us on Twitter (@TimescaleDB), so you don’t miss any of it!Time-series datais the lifeblood of the analytics revolution in nearly every industry today. One of the most difficult challenges for application developers and data scientists is aggregating data efficiently without always having to query billions (or trillions) of raw data rows. Over the years, developers and databases have created numerous ways to solve this problem, usually similar to one of the following options:DIY processes to pre-aggregate data and store it in regular tables. Although this provides a lot of flexibility, particularly with indexing and data retention, it's cumbersome to develop and maintain, particularly deciding how to track and update aggregates with data that arrives late or has been updated in the past.Extract Transform and Load (ETL) process for longer-term analytics.Even today, development teams employ entire groups that specifically manage ETL processes for databases and applications because of the constant overhead of creating and maintaining the perfect process.MATERIALIZED VIEWS.While these VIEWS are flexible and easy to create, they are static snapshots of the aggregated data. Unfortunately, developers need to manage updates using TRIGGERs or CRON-like applications in all current implementations. And in all but a very few databases, all historical data is replaced each time, preventing developers from dropping older raw data to save space and computation resources every time the data is refreshed.Most developers head down one of these paths because we learn, often the hard way, that running reports and analytic queries over the same raw data, request after request, doesn't perform well under heavy load. In truth, most raw time-series data doesn't change after it's been saved, so these complex aggregate calculations return the same results each time.In fact, as a long-term time-series database developer, I've used all of these methods too, so that I could manage historical aggregate data to make reporting, dashboards, and analytics faster and more valuable, even under heavy usage.I loved when customers were happy, even if it meant a significant amount of work behind the scenes maintaining that data.But, I always wished for a more straightforward solution.How TimescaleDB Improves Queries on Aggregated Data in PostgreSQLIn 2019, TimescaleDB introduced continuous aggregates to solve this very problem, making the ongoing aggregation of massive time-series data easy and flexible. This is the feature that first caught my attention as a PostgreSQL developer looking to build more scalable time-series applications—precisely because I had been doing it the hard way for so long.Continuous aggregates look and act like materialized views in PostgreSQL, but with many of the additional features I was looking for (if you want to learn more about views, materialized views, and continuous aggregates, check out this lesson from our Foundations of PostgreSQL and TimescaleDB course). These are just some of the things they do:Automatically track changes and additions to the underlying raw data.Provide configurable, user-defined policies to keep the materialized data up-to-date automatically.Automatically append new data (asreal-time aggregatesby default) before the scheduled process has materialized to disk. This setting is configurable.Retain historical aggregated data even if the underlying raw data is dropped.Can be compressed to reduce storage needs and further improve the performance of analytic queries.Keep dashboards and reports running smoothly.Table comparing the functionality of PostgreSQL materialized views with continuous aggregates in TimescaleDBOnce I tried continuous aggregates, I realized that TimescaleDB provided the solution that I (and many other PostgreSQL users) were looking for. With this feature, managing and analyzing massive volumes of time-series data in PostgreSQL finally felt fast and easy.What About Other Databases?By now, some readers might be thinking something along these lines:“Continuous aggregates may help with the management and analytics of time-series data in PostgreSQL, but that’s what NoSQL databases are for—they already provide the features you needed from the get-go. Why didn’t you try a NoSQL database?”Well, I did.There are numerous time-series and NoSQL databases on the market that attempt to solve this specific problem. I looked at (and used) many of them. But from my experience, nothing can quite match the advantages of a relational database with a feature like continuous aggregates for time-series data. These other options provide a lot of features for a myriad of use cases, but they weren't the right solution for this particular problem, among other things.What about MongoDB?MongoDBhas been the go-to for many data-intensive applications. Included since version 4.2 is a feature calledOn-Demand Materialized Views. On the surface, it works similar to a materialized view by combining theAggregation Pipelinefeature with a $merge operation to mimic ongoing updates to an aggregate data collection. However, there is no built-in automation for this process, and MongoDB doesn't keep track of any modifications to underlying data. The developer is still required to keep track of which time frames to materialize and how far back to look.What about InfluxDB?For many yearsInfluxDBhas been the destination for time-series applications. Althoughwe've discussed in other articles how InfluxDB doesn't scale effectively, particularly with high cardinality datasets, it does provide a feature calledContinuous Queries.This feature is also similar to a materialized view and goes one step further than MongoDB by automatically keeping the dataset updated. Unfortunately, it suffers from the same lack of raw data monitoring and doesn't provide nearly as much flexibility as SQL in how the datasets are created and stored.What about Clickhouse?Clickhouse, and several recent forks likeFirebolt, have redefined the way some analytic workloads perform. Even with some of theimpressive query performance, it provides a mechanism similar to a materialized view as well, backed by anAggregationMergeTreeengine. In a sense, this provides almost real-time aggregated data because all inserts are saved to both the regular table and the materialized view. The biggest downside of this approach is dealing with updates or modifying the timing of the process.Recent Improvements in Continuous Aggregates: Meet TimescaleDB 2.7Continuous aggregates were first introduced inTimescaleDB 1.3solving the problems that many PostgreSQL users, including me, faced with time-series data and materialized views: automatic updates, real-time results, easy data management, and the option of using the view for downsampling.But continuous aggregates have come a long way. One of the previous improvements was the introduction ofcompression for continuous aggregates in TimescaleDB 2.6. Now, we took it a step further with the arrival of TimescaleDB 2.7, which introduces dramatic performance improvements in continuous aggregates.They are now blazing fast—up to 44,000x faster in some queries than in previous versions.Let me give you one concrete example:in initial testing using live, real-time stock trade transaction data, typical candlestick aggregates were nearly 2,800x faster to querythan in previous versions of continuous aggregates (which were already fast!)Later in this post, we will dig into the performance and storage improvements introduced by TimescaleDB 2.7 by presenting a complete benchmark of continuous aggregates using multiple datasets and queries. 🔥But the improvements don’t end here.First, the new continuous aggregates also require 60 % less storage (on average) than before for many common aggregates, which directly translates into storage savings.Second, in previous versions of TimescaleDB, continuous aggregates came with certain limitations: users, for example, could not use certain functions like DISTINCT, FILTER, or ORDER BY. These limitations are now gone. TimescaleDB 2.7 ships with a completely redesigned materialization process that solves many of the previous usability issues, so you can use any aggregate function to define your continuous aggregate.Check out our release notes for all the details on what's new.✨ A big thank you to the Timescale engineers that made the improvements in continuous aggregates possible, with special mentions to Fabrízio Mello, Markos Fountoulakis, and David Kohn.And now, the fun part.Show Me the Numbers: Benchmarking Aggregate QueriesTo test the new version of continuous aggregates, we chose two datasets that represent common time-series datasets: IoT and financial analysis.IoT dataset (~1.7 billion rows):The IoT data we leveraged is the New York City Taxicab dataset that's been maintained by Todd Schneider for a number of years, and scripts are available in hisGitHub repositoryto load data into PostgreSQL. Unfortunately, a week after his latest update, the transit authority that maintains the actual datasets changed their long-standing export data format from CSV to Parquet—which means the current scripts will not work. Therefore, the dataset we tested with is from data prior to that change and covers ride information from 2014 to 2021.Stock transactions dataset (~23.7 million rows):The financial dataset we used is a real-time stock trade dataset provided byTwelve Dataand ingests ongoing transactions for the top 100 stocks by volume from February 2022 until now. Real-time transaction data is typically the source of many stock trading analysis applications requiring aggregate rollups over intervals for visualizations likecandlestick chartsand machine learning analysis. While our example dataset is smaller than a full-fledged financial application would maintain, it provides a working example of ongoing data ingestion using continuous aggregates, TimescaleDBnative compression, andautomated raw data retention(while keeping aggregate data for long-term analysis).You can use a sample of this data, generously provided by Twelve Data, to try all of the improvements in TimescaleDB 2.7 by followingthis tutorial, which provides stock trade data for the last 30 days. Once you have the database setup, you can take it a step further by registering for an API key andfollowing our tutorial to ingest ongoing transactions from the Twelve Data API.Creating Continuous Aggregates Using Standard PostgreSQL Aggregate FunctionsThe first thing we benchmarked was to create an aggregate query that used standard PostgreSQL aggregate functions likeMIN(),MAX(), andAVG(). In each dataset we tested, we created the same continuous aggregate in TimescaleDB 2.6.1 and 2.7, ensuring that both aggregates had computed and stored the same number of rows.IoT datasetThis continuous aggregate resulted in 1,760,000 rows of aggregated data spanning seven years of data.CREATE MATERIALIZED VIEW hourly_trip_stats
WITH (timescaledb.continuous, timescaledb.finalized=false) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
	avg(fare_amount) avg_fare,
	min(fare_amount) min_fare,
	max(fare_amount) max_fare,
	avg(trip_distance) avg_distance,
	min(trip_distance) min_distance,
	max(trip_distance) max_distance,
	avg(congestion_surcharge) avg_surcharge,
	min(congestion_surcharge) min_surcharge,
	max(congestion_surcharge) max_surcharge,
	cab_type_id,
	passenger_count
FROM 
	trips
GROUP BY 
	bucket, cab_type_id, passenger_countStock transactions datasetThis continuous aggregate resulted in 950,000 rows of data at the time of testing, although these are updated as new data comes in.CREATE MATERIALIZED VIEW five_minute_candle_delta
WITH (timescaledb.continuous) AS
    SELECT
        time_bucket('5 minute', time) AS bucket,
        symbol,
        FIRST(price, time) AS ""open"",
        MAX(price) AS high,
        MIN(price) AS low,
        LAST(price, time) AS ""close"",
        MAX(day_volume) AS day_volume,
        (LAST(price, time)-FIRST(price, time))/FIRST(price, time) AS change_pct
    FROM stocks_real_time srt
    GROUP BY bucket, symbol;To test the performance of these two continuous aggregates, we selected the following queries, all common queries among our users for both the IoT and financial use cases:SELECT COUNT (*)SELECT COUNT (*) with WHEREORDER BYtime_bucket reaggregationFILTERHAVINGLet’s take a look at the results.Query #1: `SELECT COUNT(*) FROM…`Doing aCOUNT(*)from PostgreSQL is a known performance bottleneck. It's one of the reasons we created theapproximate_row_count()function in TimescaleDB which uses table statistics to provide a close approximation of the overall row count. However, it's instinctual for most users (and ourselves, if we're honest) to try and get a quick row count by doing aCOUNT(*)query:-- IoT dataset
SELECT count(*) FROM hourly_trip_stats;

-- Stock transactions dataset
SELECT count(*) FROM five_min_candle_delta;And most users recognized that in previous versions of TimescaleDB, the materialized data seemed slower than normal to do a COUNT over.Thinking about our two example datasets, both continuous aggregates reduce the overall row count from raw data by 20x or more. So, while counting rows in PostgreSQL is slow, it always felt a little slower than it had to be. The reason was that not only did PostgreSQL have to scan and count all of the rows of data, it had to group the data a second time because of some additional data that TimescaleDB stored as part of the original design of continuous aggregates. With the new design of continuous aggregates in TimescaleDB 2.7, that second grouping is no longer required, and PostgreSQL can just query the data normally, translating into faster queries.Performance of a query with SELECT COUNT (*) in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7Query #2: SELECT COUNT(*) Based on The Value of a ColumnAnother common query that many analytic applications perform is to count the number of records where the aggregate value is within a certain range:-- IoT  dataset
SELECT count(*) FROM hourly_trip_stats
WHERE avg_fare > 13.1
AND bucket > '2018-01-01' AND bucket < '2019-01-01';

-- Stock transactions dataset
SELECT count(*) FROM five_min_candle_delta
WHERE change_pct > 0.02;In previous versions of continuous aggregates, TimescaleDB had to finalize the value before it could be filtered against the predicate value, which caused queries to perform more slowly. With the new version of continuous aggregates, PostgreSQL can now search for the value directly,andwe can add an index to meaningful columns to speed up the query even more!In the case of the financial dataset, we see a very significant improvement: 1,336x faster. The large change in performance can be attributed to the formula query that has to be calculated over all of the rows of data in the continuous aggregate. With the IoT dataset, we're comparing against a simple average function, but for the stock data, multiple values have to be finalized (FIRST/LAST) before the formula can be calculated and used for the filter.Performance of a query with SELECT COUNT (*) plus WHERE in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7Query #3: Select Top 10 Rows by ValueTaking the first example a step further, it's very common to query data within a range of time and get the top rows:-- IoT dataset
SELECT * FROM hourly_trip_stats
ORDER BY avg_fare desc
LIMIT 10;

-- Stock transactions dataset
SELECT * FROM five_min_candle_delta
ORDER BY change_pct DESC 
LIMIT 10;In this case, we tested queries with the continuous aggregate set to providereal-time results(the default for continuous aggregates) and materialized-only results. When set to real-time, TimescaleDB always queries data that's been materialized first and then appends (with aUNION) any newer data that exists in the raw data but that has not yet been materialized by the ongoing refresh policy. And, because it's now possible to index columns within the continuous aggregate, we added an index on theORDER BYcolumn.Performance of a query with ORDER BY in a continuous aggregate TimescaleDB 2.6.1 and TimescaleDB 2.7Yes, you read that correctly.Nearly 45,000x better performance onORDER BYwhen the query only searches through materialized data.The dramatic difference between real-time and materialized-only queries is because of theUNIONof both materialized and raw aggregate data. The PostgreSQL planner needs to union the total result before it can limit the query to 10 rows (in our example), and so all of the data from both tables need to be read and ordered first. When you only query materialized data, PostgreSQL and TimescaleDB knows that it can query just the index of the materialized data.Again, storing the finalized form of your data and indexing column values dramatically impacts the querying performance of historical aggregate data! And all of this is updated continuously over time in a non-destructive way—something that's impossible to do with any other relational database, including vanilla PostgreSQL.Query #4: Timescale Hyperfunctions to Re-aggregate Into Higher Time BucketsAnother example we wanted to test was the impact finalizing data values has on our suite ofanalytical hyperfunctions. Many of the hyperfunctions we provide as part of theTimescaleDB Toolkitutilize custom aggregate values that allow many different values to be accessed later depending on the needs of an application or report. Furthermore, these aggregate values can bere-aggregated into different size time buckets. This means that if the aggregate functions fit your use case, one continuous aggregate can produce results for many different time_bucket sizes! This is a feature many users have asked for over time, and hyperfunctions make this possible.For this example, we only examined the New York City Taxicab dataset to benchmark the impact of finalized CAGGs. Currently, there is not an aggregate hyperfunction that aligns with the OHLC values needed for the stock data set, however,there is a feature requestfor it! (😉)Although there are not currently any one-to-one hyperfunctions that provide exact replacements for our min/max/avg example, we can still observe the query improvement using atdigestvalue for each of the columns in our original query.Original min/max/avg continuous aggregate for multiple columns:CREATE MATERIALIZED VIEW hourly_trip_stats
WITH (timescaledb.continuous, timescaledb.finalized=false) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
	avg(fare_amount) avg_fare,
	min(fare_amount) min_fare,
	max(fare_amount) max_fare,
	avg(trip_distance) avg_distance,
	min(trip_distance) min_distance,
	max(trip_distance) max_distance,
	avg(congestion_surcharge) avg_surcharge,
	min(congestion_surcharge) min_surcharge,
	max(congestion_surcharge) max_surcharge,
	cab_type_id,
	passenger_count
FROM 
	trips
GROUP BY 
	bucket, cab_type_id, passenger_countHyperfunction-based continuous aggregate for multiple columns:CREATE MATERIALIZED VIEW hourly_trip_stats_toolkit
WITH (timescaledb.continuous, timescaledb.finalized=false) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
	tdigest(1,fare_amount) fare_digest,
	tdigest(1,trip_distance) distance_digest,
	tdigest(1,congestion_surcharge) surcharge_digest,
	cab_type_id,
	passenger_count
FROM 
	trips
GROUP BY 
	bucket, cab_type_id, passenger_countWith the continuous aggregate created, we then queried this data in two different ways:1. Using the same `time_bucket()` size defined in the continuous aggregate, which in this example was one-hour data.SELECT 
	bucket AS b,
	cab_type_id, 
	passenger_count,
	min_val(ROLLUP(fare_digest)),
	max_val(ROLLUP(fare_digest)),
	mean(ROLLUP(fare_digest))
FROM hourly_trip_stats_toolkit
WHERE bucket > '2021-05-01' AND bucket < '2021-06-01'
GROUP BY b, cab_type_id, passenger_count 
ORDER BY b DESC, cab_type_id, passenger_count;Performance of a query with time_bucket() in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7 (the query uses the same bucket size as the definition of the continuous aggregate)2. We re-aggregated the data from one-hour buckets into one-day buckets.This allows us to efficiently query different bucket lengths based on the original bucket size of the continuous aggregate.SELECT 
	time_bucket('1 day', bucket) AS b,
	cab_type_id, 
	passenger_count,
	min_val(ROLLUP(fare_digest)),
	max_val(ROLLUP(fare_digest)),
	mean(ROLLUP(fare_digest))
FROM hourly_trip_stats_toolkit
WHERE bucket > '2021-05-01' AND bucket < '2021-06-01'
GROUP BY b, cab_type_id, passenger_count 
ORDER BY b DESC, cab_type_id, passenger_count;Performance of a query with time_bucket() in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7. The query re-aggregates the data from one-hour buckets into one-day bucketsIn this case, the speed is almost identical because the same amount of data has to be queried. But if these aggregates satisfy your data requirements, only one continuous aggregate would be necessary in many cases, rather than a different continuous aggregate for each bucket size (one minute, five minutes, one hour, etc.)Query #5: Pivot Queries With FILTERIn previous versions of continuous aggregates, many common SQL features were not permittedbecause of how the partial data was stored and finalized later. Using a PostgreSQLFILTERclause was one such restriction.For example, we took the IoT dataset and created a simpleCOUNT(*)to calculate each company's number of taxi rides (cab_type_id) for each hour. Before TimescaleDB 2.7, you would have to store this data in a narrow column format, storing a row in the continuous aggregate for each cab type.CREATE MATERIALIZED VIEW hourly_ride_counts_by_type 
WITH (timescaledb.continuous, timescaledb.finalized=false) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
	cab_type_id,
  	COUNT(*)
FROM trips
  	WHERE cab_type_id IN (1,2)
GROUP BY 
	bucket, cab_type_id;To then query this data in a pivoted fashion, we couldFILTERthe continuous aggregate data after the fact.SELECT bucket,
	sum(count) FILTER (WHERE cab_type_id IN (1)) yellow_cab_count,
  	sum(count) FILTER (WHERE cab_type_id IN (2)) green_cab_count
FROM hourly_ride_counts_by_type
WHERE bucket > '2021-05-01' AND bucket < '2021-06-01'
GROUP BY bucket
ORDER BY bucket;In TimescaleDB 2.7, you can now store the aggregated data using aFILTERclause to achieve the same result in one step!CREATE MATERIALIZED VIEW hourly_ride_counts_by_type_new 
WITH (timescaledb.continuous) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
  	COUNT(*) FILTER (WHERE cab_type_id IN (1)) yellow_cab_count,
  	COUNT(*) FILTER (WHERE cab_type_id IN (2)) green_cab_count
FROM trips
GROUP BY 
	bucket;Querying this data is much simpler, too, because the data is already pivoted and finalized.SELECT * FROM hourly_ride_counts_by_type_new 
WHERE bucket > '2021-05-01' AND bucket < '2021-06-01'
ORDER BY bucket;This saves storage (50 % fewer rows in this case) and CPU to finalize theCOUNT(*)and then filter the results each time based oncab_type_id. We can see this in the query performance numbers.Performance of a query with FILTER in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7.Being able to useFILTERand other SQL features improve both developer experience and flexibility long term!Query #6: HAVING Stores Significantly Less Materialized DataAs a final example of how the improvements to continuous aggregates will impact your day-to-day development and analytics processes, let's look at a simple query that uses aHAVINGclause to reduce the number of rows that the aggregate stores.In previous versions of TimescaleDB, the having clause couldn't be applied at materialization time. Instead, theHAVINGclause was applied after the fact to all of the aggregated data as it was finalized. In many cases, this dramatically affected both the speed of queries to the continuous aggregate and the amount of data stored overall.Using our stock data as an example, let's create a continuous aggregate that only stores a row of data if thechange_pctvalue is greater than 20 %. This would indicate that a stock price changed dramatically over one hour, something we don't expect to see in most hourly stock trades.CREATE MATERIALIZED VIEW one_hour_outliers
WITH (timescaledb.continuous) AS
    SELECT
        time_bucket('1 hour', time) AS bucket,
        symbol,
        FIRST(price, time) AS ""open"",
        MAX(price) AS high,
        MIN(price) AS low,
        LAST(price, time) AS ""close"",
        MAX(day_volume) AS day_volume,
        (LAST(price, time)-FIRST(price, time))/LAST(price, time) AS change_pct
    FROM stocks_real_time srt
    GROUP BY bucket, symbol
   HAVING (LAST(price, time)-FIRST(price, time))/LAST(price, time) > .02;Once the dataset is created, we can query each aggregate to see how many rows matched our criteria.SELECT count(*) FROM one_hour_outliers;Performance of a query with HAVING in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7The biggest difference here (and the one that will more negatively impact the performance of your application over time) is the storage size of this aggregated data. Because TimescaleDB 2.7 only stores rows that meet the criteria, the data footprint is significantly smaller!Storage footprint of a continuous aggregate bucketing stock transactions by the hour in TimescaleDB 2.6.1 and TimescaleDB 2.7Storage Savings in TimescaleDB 2.7One of the final pieces of this update that excites us is how much storage will be saved over time. On many occasions, users with large datasets that contained complex equations in their continuous aggregates would join ourSlack communityto ask why more storage is required for the rolled-up aggregate than the raw data.In every case we've tested, the new, finalized form of continuous aggregates is smaller than the same example in previous versions of TimescaleDB, with or without aHAVINGclause that might filter additional data out.Storage savings for different continuous aggregates in TimescaleDB 2.6.1 and TimescaleDB 2.7The New Continuous Aggregates Are a Game-ChangerFor those dealing with massive amounts of time-series data, continuous aggregates are the best way to solve a problem that has long haunted PostgreSQL users. The following list details how continuous aggregates expand materialized views:They always stay up-to-date, automatically tracking changes in the source table for targeted, efficient updates of materialized data.You can use configurable policies to conveniently manage refresh/update interval.You can keep your materialized data even after the raw data is dropped, allowing you to downsample your large datasets.And you can compress older data to save space and improve analytic queries.And in TimescaleDB 2.7, continuous aggregates got much better. First, they are blazing fast: as we demonstrated with our benchmark, the performance of continuous aggregates got consistently better across queries and datasets, up to thousands of times better for common queries. They also got lighter, requiring an average of 60 % less storage.But besides the performance improvements and storage savings, there are significantly fewer limitations on the types of aggregate queries you can use with continuous aggregates, such as:Aggregates with DISTINCTAggregates with FILTERAggregates with FILTER in HAVING clauseAggregates without combine functionOrdered-set aggregatesHypothetical-set aggregatesThis new version of continuous aggregates is available by default inTimescaleDB 2.7: now, when you create a new continuous aggregate, you will automatically benefit from all the latest changes.Read our release notes for more information on TimescaleDB 2.7, and for instructions on how to upgrade,check out our docs.Looking to migrate your existing continue aggregates to the new version? Now, with TimescaleDB 2.8.1, you don’t have to worry aboutmigrating from the old continuous aggregates to the new. Say hello to our frictionless migration, an in-place upgrade that avoids disrupting queries over continuous aggregates in applications and dashboards and every time the data is not in the original hypertable.☁️🐯 Timescale Cloud avoids the manual work involved in updating your TimescaleDB version. Updates take place automatically during a maintenance window picked by you.Learn moreabout automatic version updates in Timescale Cloud, and to test if yourself,start a free trial!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-we-made-data-aggregation-better-and-faster-on-postgresql-with-timescaledb-2-7/
2023-10-31T12:42:41.000Z,New Timescale Enterprise Tier: A Solution for Mature Applications,"Timescale is excited to announce the launch of our new Enterprise Tier, a comprehensive set of features designed to meet the needs of enterprise customers using the Timescale database platform.Timescale is a mature cloud platform that scales PostgreSQL’s performance for demanding applications, including time-series, analytics, and more.  Businesses ranging from growing startups to large enterprises trust Timescale when building products on their sensor data, application data, financial data, and more.As we highlighted in ourCloud Launch Week kickoff announcement, Timescale offers best-in-class performance for PostgreSQL, easy scalability, high availability and reliability, automated upgrades and software patching, security, deep observability, cost-effective pricing, global reach, and top-notch support.With its new Enterprise Tier, Timescale is even better suited for businesses with mature applications that require high performance, scalability, security, and compliance guarantees.Here is a summary of the key features and benefits of the Timescale Enterprise Tier:Improved security and compliance:The Enterprise Tier includes SAML/SSO support, SOC 2 Type 2 certification with annual reports, custom security questionnaire support, and a 99.9% SLA with financial commitments. This ensures that your Timescale deployments are secure, compliant, and always available.Enablement and reduced costs:The Enterprise Tier includes compression assistance and migration assistance, as well as quarterly architectural reviews to optimize usage and performance. These services can help you to reduce your costs and save time when migrating your data to Timescale.Data protection and recovery:The Enterprise Tier includes 14-day Point-In-Time Recovery (PITR), which can be fully controlled through the Timescale console. Knowing that your data is always protected and recoverable gives you peace of mind.Production-level support:The Enterprise Tier includes ourProduction-Level Support packagefrom a team of dedicated experts who can help you with any Timescale-related issue in our mature cloud platform.Peace of mind:With the Enterprise Tier, you can be confident that your Timescale database deployments are secure, compliant, performant, scalable, and supported by a team of experts.How to get startedIf you want to learn more about the Enterprise Tier or sign up for a free trial,contact us. We’ll be happy to assist and answer all your questions!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/new-timescale-enterprise-tier/
2021-10-11T13:02:56.000Z,"What Are Traces, and How SQL (Yes, SQL) and OpenTelemetry Can Help Us Get More Value Out of Traces to Build Better Software","⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Developing software is hard. Debugging complex software systems is often harder. When our software systems are not healthy, we need ways to quickly identify what is happening, and then dig deeper to understand why it is happening (and then fix the underlying issues). But in modern architectures based on microservices, Kubernetes and cloud, identifying problems (let alone predicting them) has become more and more difficult.Enter observability, built on the collection of telemetry from modern software systems. This telemetry typically comes in the form of three signals: metrics, logs, and traces.Metrics and logs are well known and have been widely adopted for many years through tools likeNagios,Prometheus, or theELKstack.Traces, on the other hand, are relatively new and have seen much lower adoption. Why? Because, for most engineers, tracing is still a relatively new concept. Because getting started takes a lot of manual instrumentation work. And because, once we have done that work,getting value out of trace datais hard (for example, most tracing tools today just provide the ability to look up a trace by id or apply very simple filtering like Jaeger or Grafana Tempo).However, traces are key to understanding the behavior of and troubleshooting modern architectures.Here we demystify traces and explain what they are, and why they are useful, using a concrete example. Then we describe OpenTelemetry and explain how it vastly simplifies the manual instrumentation work required to generate traces.Finally, we announce the beta release of trace support in Promscale, the observability backend powered by SQL, via OpenTelemetry, and describe how Promscale and SQL enable us to get much more value out of trace data.Read on for more. If you’d like to get started with Promscale right away:Install the latest version of Promscale, following the instructions in ourGitHub repository(appreciate any GitHub stars!). As a reminder, Promscale is open-source and completely free to use.Join the TimescaleDB Slack community, where you’ll find 7K+ developers and Timescale engineers active in all channels. (The dedicated #promscale channel has 2.9K+ members, so it’s a great place to connect with like-minded community members, ask questions, get advice, and more).If you would like to connect with us, catch our talks atPromCon North America. We are also atKubeCon+CloudNativeCon North America🎉 Come to say hi to the Timescale booth (#S76)or join the Timescale virtual booth to chat with Promscale engineers, see Promscale in action during Office Hours session (Oct 14, 10:30 am PDT), and get cool swag!What Is a Trace?A “trace” represents how an individual request flows through the various microservices in a distributed system. A “span” is the fundamental building block of a trace, representing a single operation. In other words, a trace is a collection of underlying spans.Let’s look at a specific example to illustrate the concept.Imagine that we have a news site that is made of four micro-services:Thefrontendservice, which serves the website to our customer’s browser.Thenewsservice, which provides the list of articles from our news database that is populated by the editorial system (and has the ability to search articles and get individual articles together with their comments).Thecommentservice, which lets you save new comments in the comment database and retrieve all comments for an article.Theadvertisingservice, which returns a list of ads to display from a third-party provider.An architecture diagram for an example news site made up of four microservices: a frontend service, a news service, a comment service, and an advertising service.When a user clicks on the link to view an article on her browser this is what happens:1) Thefrontendservice receives the request.2) Thefrontendservice calls thenewsservice passing the identifier for the article.3) Thenewsservice calls the news database to retrieve the article.4) Thenewsservice calls thecommentservice to retrieve the comments for the  article.5) Thecommentservice calls the comment database and runs a query to retrieve all comments for the article.6) Thenewsservice gets all comments for the article and sends the article and the comments back to thefrontendservice.7) Thefrontendservice calls theadvertisingservice passing the contents of the article.8) Theadvertisingservice makes a REST API request to the third-party ads provider with the contents of the article to retrieve the list of optimized ads to display.9) Thefrontendservice builds the HTML page including the news, the comments and the ads send the response back to the user’s browser.When adding trace instrumentation to each of those services, you generate spans for each operation in the execution of the request outlined above (and more if you want more detailed tracking). This would result in a hierarchy of spans like shown in the diagram below:Hierarchy of spans for the news site microservice architecture. The length of each span indicates their duration.The entry point for the request is thefrontendservice. The first span thefrontendservice emits covers the entire execution of the request. That span is called the root span. All other spans are descendants of that root span. The length of each span indicates their duration.What do traces show?As you can see from the diagram above there are two key pieces of information we can extract from a trace:A connected representation of all the steps to process individual requests which very easily allow us to zero-in on the service and operation that is causing issues in your system when troubleshooting a production problem.The dependencies between different components in the system which we could use to build a map of how those components connect to each other. In a large system with tens or hundreds of components it is important to explore and understand the topology of the system to identify problems and improvements.Faster troubleshooting with tracesLet’s see through a practical example of how traces help identify problems in your applications faster.Continuing with our example, imagine that less than 1% of the REST API requests to the ads provider are suffering from slow response time (30 seconds). But those requests always come from the same set of users who are complaining to your support team and on Twitter.We look into the problem but aggregate percentile metrics (see our blog post onpercentile metricsto learn how to use them) are not indicating any problem because overall performance (99th percentile of response time, p99 for short) is great since this only impacts a small number of requests.Then we start looking into our logs but that’s a daunting task. We have to look at logs for each individual service and there are concurrent requests that make it very hard to read and connect those logs to identify the problem.What we need is to reconstruct requests around the time users reported the issue, find the requests that were slow, and then determine where that slowness happened.With logs, we have to do that manually which for a high traffic site it would take hours to do.With traces, we can do that automatically and quickly identify where the problem is.One simple way to do it is to search for the slowest traces, that is, top ten root spans with the highest duration during the time the users complained and look at what’s consuming most of the execution time. In our example, by looking at the visual trace representation we would very quickly see that what’s common in the slowest traces is that the request to the ads provider REST API is taking too long. An even better way (if your tracing system allows for that) would be to run a query to retrieve the slowest spans in the execution path of the slowest traces which would return the spans tracking the REST API calls.Note that looking at the p99 response time of those REST API calls would not have revealed any problem either because the problem occurred in less than 1% of those requests.We’ve quickly narrowed down where the problem is and can start looking for a solution as well as inform the ads provider of the issue so they can investigate it.One quick fix could be to put in place a one-second timeout in the API request so that a problem with the ads provider doesn’t impact your users. Another more sophisticated solution could be to make ads rendering an asynchronous call so it doesn’t impact rendering the article.Traces help us proactively make software betterWe can also use traces to proactively make our software better. For example, we could search for the slowest spans that represent database requests to identify queries to optimize. We could search for requests (i.e., traces) that involve a high number of services, or many database calls, or too many external service calls, and look for ways to optimize them and/or simplify the architecture of the system.RecapWe have seen how traces are useful to troubleshoot problems in microservices environments faster and discover improvements that we could implement to help us build better software that delights our customers.So what’s needed to get the benefits that traces provide? A tool to instrument services to generate traces, and a system to store, query and visualize them.Continue reading to learn how OpenTelemetry makes instrumentation easier, and how Promscale and SQL enable us to get more value out of traces, faster.Traces in OpenTelemetryOpenTelemetryis a vendor-agnostic emerging standard to instrument and collect traces (and also metrics and logs!) that can then be sent to and analyzed in any OpenTelemetry compatible backend. It has recently beenaccepted as a CNCF incubating projectand it has a lot of momentum: it’sthe second most active CNCF project, only after Kubernetes, with contributions fromall major observability vendors, cloud providers, and many end users(including Timescale).OpenTelemetry includes a number of core components:API specification: defines how to produce telemetry data in the form of traces, metrics, and logs.Semantic conventions: defines a set of recommendations to standardize the information to include in the telemetry (for example attributes like the status code of a span representing an http request) and ensure better compatibility between systems.OpenTelemetry protocol (OTLP): defines a standard encoding, transport, and delivery mechanism of telemetry data between the different components of an observability stack: telemetry sources, collectors, backends, etc.SDKs: language-specific implementations of the OpenTelemetry API with additional capabilities like processing and exporting for traces, metrics and logs.Instrumentation libraries: language-specific libraries that provide instrumentation for other libraries. All instrumentation libraries support manual instrumentation and several offer automatic instrumentation through byte-code injection.Collector: this component provides the ability to receive telemetry from a wide variety of sources and formats, process it and export it to a number of different backends. It eliminates the need to manage multiple agents and collectors.An architecture diagram illustrating the core components of the OpenTelemetry collector, the inputs it accepts and possible outputs it can produce.To collect traces from our code with OpenTelemetry we will use the SDKs and instrumentation libraries for the language your services are written in. Instrumentation libraries make OpenTelemetry easy to adopt because they can auto-instrument (yes, auto-instrument!) our code for services written in languages that allow for injecting instrumentation at runtime like Java and Node.js.For example, the OpenTelemetry Java instrumentation library automatically collects traces from alarge number of libraries and frameworks. For languages where that is not possible (like Go) we also get libraries that simplify the instrumentation but require more changes to the code.Even more, developers of libraries and components are already adding OpenTelemetry instrumentation directly in their code. Two examples of that areKubernetesandGraphQL Apollo Server.Once our code is instrumented we have to configure the SDK to export the data to an observability backend. While we can send the data directly from our application to a backend, it is more common to send the data to the OpenTelemetry Collector and then have it send the data to one or multiple backends. This way you can simplify the management and configuration of where you want to send the data and have the possibility to do additional processing (downsampling, dropping, transforming data) before it is sent to another system for analysis.Anatomy of an OpenTelemetry traceTheOpenTelemetry tracing specificationdefines the data model for a trace. Technically, a trace is adirected acyclic graphof Spans with parent-child relationships. Traces must include an operation name, start and finish timestamps, a parent Span identifier and the SpanContext. The SpanContext contains the TraceId, the SpanID, TraceFlags and the Tracestate. Optionally, Spans can also have a list of Events, Links to other related Spans and a set of Attributes (key-value pairs). Events are typically used to add to a Span one-time events like errors (error message, stacktrace and error code) or log lines that were recorded during the span execution. Links are less common but allow OpenTelemetry to support special scenarios like relating spans that are included in a batch operation with the batch operation span which would be initiated by multiple parents (i.e., all the individual spans that add elements to the batch).Attributes can contain any key-value pair.Trace semantic conventionsdefine some mandatory and optional attributes to help with interoperability. For example, a span representing the client or server of anHTTP requestmust have an http.method attribute and may have http.url and http.host attributes.Data model of an OpenTelemetry SpanRecapOpenTelemetry is a new vendor-agnostic standard that makes trace instrumentation and collection much easier through automation, SDKs and a protocol to provide interoperability with observability tools.Analyzing Traces With SQLAbove, we talked about several ways to get value out of traces. For example, when debugging our news application, searching for the top 10 root spans with the highest duration during the time that users complained. Or, when trying to proactively improve our news application, to search for the slowest spans that represent database requests to identify queries to optimize. Or similarly, to search for requests (i.e. traces) that involve a high number of services, or many database calls, or too many external service calls, and look for ways to optimize them and/or simplify the architecture of the system.Yet there is no standard way to analyze trace data to ask these questions. In the past, a small number of organizations (e.g., Google, Twitter, Uber, etc.) have built their own systems to do deeper analysis on their trace data. There have also been a number of open-source efforts (e.g., Jaeger, Zipkin, Grafana Tempo) which provide a UI that is very helpful to find and visualize individual traces but don’t provide the flexibility of a language to run any query and aggregate traces as needed to derive new insights.Turns out that we already have a universal query language for data analysis, one that most developers (and many non-developers) already know: SQL.In fact, with SQL, we can interrogate our trace data to answer any question we need to answer, in a way not possible with existing open-source tracing tools like Jaeger and Grafana Tempo.For example:List the operations with the highest error rates.List the slowest API methods.List the slowest database queries across all services.Identify customers suffering from the worst database query performance and how it compares to performance across all customers.Identify the upstream service causing the load on a service that is seeing elevated load.Even more, with SQL you could correlate traces and metrics at query time. For example, you could show response time per service correlated with CPU and memory consumption per service if you have container metrics in the same database.But of course, before we can use SQL to query traces (and metrics), we need to be able to store those traces in a scalable system that supports SQL, yet is designed for observability.Enter Promscale.Promscale Is the Observability Backend Powered by SQLPromscale was first announcedone year agoas an analytical platform for Prometheus metrics. Promscale is built on the solid foundation of TimescaleDB and PostgreSQL, and therefore has full SQL support(along with many other neat featureslike 100% PromQL compliance). Today, companies like Digital Ocean, Electronic Arts and Catalytic rely on Promscale to scale Prometheus to do long-term retention and analysis on their metrics.Our vision for Promscale is to enable engineers to store all observability data (metrics, logs, traces, metadata, and other future data types) in a single mature and scalable store and analyze it through a unified and complete SQL interface that provides developers with:Broad support for observability standards (e.g., OpenTelemetry, Prometheus/PromQL, StatsD, Jaeger, Zipkin, etc.) to simplify integration in any environment.Operational simplicity with just one storage system that is easy to deploy, manage, scale, and learn about.A familiar experience with PostgreSQL as the foundation and unified access to all data via full SQL support so they don’t need to learn other query languages.Unparalleled insights through the power of TimescaleDB’s advanced time-series analytical functions and Postgres’ SQL query capabilities (joins, sub-queries, etc.) to analyze and correlate observabilityandbusiness data.100s of out-of-the-box integrations through the PostgreSQL ecosystem: visualization tools, IDEs, ORMs, management tools, performance tuning, etc.Today we are announcing the beta release of trace support in Promscale, our second major step in fulfilling our vision.Promscale exposes an ingest endpoint that is OTLP-compliant which makes integration with OpenTelemetry instrumentation straightforward. Other tracing formats like Jaeger, Zipkin or OpenCensus can also be sent to Promscale through the OpenTelemetry Collector. Traces stored in Promscale can be queried with full SQL and visualized using Jaeger or Grafana.Architecture diagram illustrating Promscale architecture, with inputs from Prometheus metrics and OpenTelemetry traces, TimescaleDB as the core database powering Promscale and outputs to a variety of tools including Jaeger and Grafana.Designing an optimized schema for tracesPromscale stores OpenTelemetry traces using the following schema:Promscale schema for tracesThe schema is heavily influenced by the OpenTelemetryprotocol buffer definitions.The heart of the model is the span table. The span table is ahypertable, using the power of TimescaleDB to manage the ingestion and querying of spans over time. Similarly, both links and events are hypertables. Link tables use the start time of the source span, whereas each event has its own associated time.The span table is self-referencing. All the spans in a given trace form a tree. The span table uses the adjacency model to capture these trees - each span contains the id of the span’s parent. The root span of each trace has aparent_span_idof 0. The OpenTelemetry protocol buffers already utilize an adjacency model to represent the tree structure, so keeping this same model at the database layer makes ingestion easier. Other models such as the path enumeration model can be derived from an adjacency model, and we do this in several convenience functions.Spans, links, events, and resources can all be decorated with zero or moreattributes. Attributes are key-value pairs in which the key is a text name, and the value can be a primitive type (string, boolean, double, or 64 bit integer), or a homogeneous array.We have chosen to use “tag” instead of “attribute” in our model. “Attribute” gets to be a real pain to type over and over and over, and by our estimation “tag” is more widely used and understood in the industry.Tag values map very closely to json, and therefore we have chosen to store tag values in thejsonb data type. PostgreSQL has extensive support for storing andmanipulatingjsonb values, includingindexingandjsonpath querying. By using jsonb, we can piggy-back on PostgreSQL’s features to provide rich ways of filtering spans that in many cases will be indexed operations.Many tags will be repeated again and again across many spans. For this reason, we are not storing tags directly in the span table, but normalizing them out intotag_keyand tag tables. This eliminates data duplication and greatly reduces the storage required for the span table.Traditionally, a many-to-many relationship between spans and tags would be represented with an additional mapping table. We decided to eschew this approach. An additional mapping table would have considerably more records than the span table itself. It would need to contain both thetrace_idandspan_idwhich are 128 bits and 64 bits respectively, a timestamptz (it would need to be a hypertable in its own right), and the 64 bit ids of the tag andtag_key. This would require significant additional storage.Instead of this mapping table, we created a domain over jsonb called a “tag_map”. Atag_mapis a json object where keys are the ids of thetag_keytable, and the associated values are ids of the tag table. Thus, thetag_mapis a set of key-value pairs oftag_keysto tag values. We have thus essentially collapsed the mapping table into the span table.The tag table may in some setups become quite large. We have utilized PostgreSQL’sdeclarative partitioningto hash partition the tag table on the key text over 64 partitions. This effectively “load balances” the tag values over many smaller tables rather than storing them all in one big table. By hash partitioning on key text, all the values for a given key will be colocated in the same partition, and thereby belong to the same partition-level indexes, improving performance. Partitioning the tags should also provide some resiliency, in that operations on one key should not impact all keys. The number 64 was chosen more or less randomly. We intend to do some testing to find an optimal number of partitions.We are sharing how our schema works to help other developers in the community looking at solving a similar problem. Knowing the design of the underlying schema can also be helpful in situations where you may need to improve the performance of some query. However, you don’t really need (and we don’t expect you!) to understand how our schema works because Promscale comes with out-of-the-box views, functions and operators that make querying the data easier and faster that we cover in the next section.Querying Traces using SQLPromscale provides unified access to all your traces and metrics through a single, robust and well-known query language: SQL. Thanks to the power of SQL, the out-of-the-box views, functions and operators built-in in Promscale and the time-series analytical functions provided by TimescaleDB you can interrogate your data about pretty much any question you need to answer.Everything you need to query traces is in theps_tracedatabase schema:3 views: span, event and link. Those views automatically join the different tables in the schema to provide a consolidated view of a span, an event or a link as if all attributes were stored in the same table.A number of operators so you can easily apply conditions to resource, span, link and event attributes (we call those attributes tags in our data model).Functions to easily navigate through and retrieve tag values.To understand how to query traces we will use some of the examples we listed at the beginning of this section. The data we use in the examples comes from a Kubernetes cluster runningHoneycomb’s forkofGoogle’s microservices demothat uses OpenTelemetry instrumentation.In the examples below we show the raw results from the SQL queries. Those results can be easily displayed in Grafana dashboards by connecting to the underlying TimescaleDB/PostgreSQL database using Grafana’s PostgreSQL datasource:Grafana dashboard showing performance metrics from querying traces in Promscale with SQLQuery 1:  List the top operations with the highest error rate in the last hourResponse time, throughput and error rate are the key metrics used to assess the health of a service. In particular, high error rate and response time are key indicators of the experience we are delivering to our users and need to be tracked closely. In this example, we look at how we can use OpenTelemetry traces and SQL to identify the main sources of errors in our services.Every span has an attribute that indicates the name of the service (service_name) and an attribute that indicates the name of the operation (name). Technically, the service name in OpenTelemetry is not a span attribute but a resource attribute. For the purposes of querying traces in Promscale we can think of service name as a span attribute as well.Every span has another attribute that indicates whether the span resulted in an error or not:status_code. If there is an error, the value ofstatus_codeiserror.Our goal is to write a query that will return one row per individual operation with the total number of executions (number of spans), total number of executions that led to an error (spans where the status code is an error) and the percentage of spans with an error for the top 10 operations with more errors:SELECT
    service_name,
    span_name as operation,
    COUNT(*) FILTER (WHERE status_code = 'STATUS_CODE_ERROR') as spans_with_error,
    COUNT(*) as total_spans,
    TO_CHAR(100*(CAST(COUNT(*) FILTER (WHERE status_code = 'STATUS_CODE_ERROR') AS float) / count(*)), '999D999%') as error_rate
FROM span
WHERE
     start_time > NOW() - INTERVAL '1 hour'
GROUP BY service_name, operation
ORDER BY error_rate DESC
LIMIT 10;As you can see, the query uses the standard SQL syntax we are all familiar with:SELECT,FROM,WHERE,GROUP BY,ORDER BY.In theSELECTclause we project the service name and the operation (which corresponds to the span name). These are also the two attributes we use to aggregate the results in theGROUP BYclause.The other three columns we project are the number of spans with an error, the total number of spans and the error rate within each error group. To do it we use some nifty SQL capabilities available in PostgreSQL:COUNT (*)which returns the total number of rows, spans in this case, in a group.COUNT (*) FILTERwhich returns the total number of spans in a group matching a certain criteria. In our case we want spans with an error which are indicated by the valueerrorin thestatus_codeattribute.CASTto convert the number of spans with error to a floating point number so when it’s divided by the total count of spans it returns a floating number and not an integer. If we don’t do this then the number will be converted to the closest integer which will be 0 since that division is always less than 0.TO_CHARto convert the error rate to an easy to read and understand percentage number.Below is an example of results from this query:service_name |             operation             | spans_with_error | total_spans | error_rate
--------------+-----------------------------------+------------------+-------------+------------
 frontend     | /cart/checkout                    |               12 |         345 |    3.478%
 frontend     | hipstershop.AdService/GetAds      |               60 |        5214 |    1.115%
 adservice    | hipstershop.AdService/GetAds      |                1 |        5214 |     .019%
 cart         | grpc.health.v1.Health/Check       |                0 |         707 |     .000%
 cart         | hipstershop.CartService/EmptyCart |                0 |         345 |     .000%
 cart         | hipstershop.CartService/GetCart   |                0 |        7533 |     .000%
 checkout     | SQL SELECT                        |                0 |         361 |     .000%
 checkout     | grpc.health.v1.Health/Check       |                0 |         718 |     .000%
 checkout     | getDiscounts                      |                0 |         345 |     .000%
 checkout     | hipstershop.CartService/EmptyCart |                0 |         345 |     .000%Which immediately indicates that we need to take a closer look at the code behind the/cart/checkoutoperation since it has a very high error rate most likely leading to many lost sales!Another thing that drives our attention in these results are the second and third rows. What’s surprising about them is that the error rate on the client side of the request(frontend - hipstershop.AdService/GetAds) is much higher than the error rate on the server side of the same request (adservice - hipstershop.AdService/GetAds). So the adservice is successfully completing the request but something is happening when transferring the response back to the frontend service.Before we move on to the next example, let’s look at a different way to write the query that leverages additional SQL capabilities. To calculate the error rate in the query above we are using twice the functions required to count spans with error and total spans. This can lead to inconsistencies if we update the query to change the way we calculate the error rate but don’t apply those changes everywhere. To avoid that we can use SQL subqueries:SELECT
  service_name,
  operation,
  spans_with_error,
  total_spans,
  TO_CHAR(100*(CAST(spans_with_error AS float)) / total_spans, '999D999%') as error_rate
FROM (
    SELECT
        service_name,
        span_name as operation,
        COUNT(*) FILTER (WHERE status_code = 'error') as spans_with_error,
        COUNT(*) as total_spans
    FROM span
    WHERE
        start_time > NOW() - INTERVAL '1 hour'
    GROUP BY service_name, operation
) AS error_summary
ORDER BY error_rate DESC
LIMIT 10;This query first builds a dataset (error_summary) with service name, operation, spans with error and total spans and then it uses the values calculated in that dataset to compute the error rate avoiding the duplication in the initial query. This change doesn’t impact the performance of the query.This is a straightforward example of what you can do with subqueries. SQL subqueries provide a lot of flexibility for analyzing your traces to derive new insights.An additional consideration is that the query searches across all spans and depending on the environment and the amount of instrumentation you could see some duplicative results because of parent-child relationships between spans. In those cases it could be better to start by looking at traces (complete request) that had an error. You can do that by only searching across root spans by adding an additional condition to the where clause:WHERE start_time > NOW() - INTERVAL '1 hour' AND parent_span_id = 0Query 2: List the top slowest operations in the last hourAt the beginning of the previous example we identified response time and error rate as two key indicators of the experience we deliver to our users. Let’s see now how we can quickly determine bottlenecks in our services.All spans contain a duration attribute that indicates how long it took to execute that span. To analyze the duration we will look at several statistics and in particular percentiles. To learn more about percentiles and why you should use them instead of averages take a look atour blog post on this subject.In this example, our goal is to write a query that will return one row per individual operation with several percentiles (99.9th, 99th, 95th) and the average of the duration in milliseconds. For this one we will search across root spans so we see which user requests have the worse response time:SELECT
    service_name,
    span_name as operation,
    ROUND(approx_percentile(0.999, percentile_agg(duration_ms))::numeric, 3) as duration_p999,
    ROUND(approx_percentile(0.99, percentile_agg(duration_ms))::numeric, 3) as duration_p99,
    ROUND(approx_percentile(0.95, percentile_agg(duration_ms))::numeric, 3) as duration_p95,
    ROUND(avg(duration_ms)::numeric, 3) as duration_avg
FROM span
WHERE
    start_time > NOW() - INTERVAL '1 hour' AND
    parent_span_id = 0
GROUP BY service_name, operation
ORDER BY duration_p99 DESC
LIMIT 10;This query uses TimescaleDB’sapprox_percentilehyperfunctionto calculate the different percentiles. You could write the same query using PostgreSQL’s nativeprecentile_contfunction. However,approx_percentileis faster thanpercentile_contwhile incurring a small error (less than 3%). With our test data, approx_percentile performs 35% faster thanpercentile_contwith minimal error (less than 3%). If you need more precision, you can decrease the error at the expense of lower performance by replacingpercentile_aggwithuddsketch.We use theROUNDfunction to limit the number of decimals shown in each column. We need to use::numericto convert the return value ofapprox_percentilewhich is double precision to a type supported by theROUNDfunction, numeric in this case.service_name |              operation              | duration_p99 | duration_p999 | duration_p95 | duration_avg
--------------+-------------------------------------+--------------+---------------+--------------+--------------
 frontend     | /cart/checkout                      |    20658.359 |     20658.359 |     1238.359 |      600.609
 adservice    | AdService.start                     |    13319.743 |     13319.743 |    13319.743 |    13307.008
 cart         | grpc.health.v1.Health/Check         |     1843.628 |      1843.628 |      911.864 |      132.689
 frontend     | /                                   |     1159.651 |      1159.651 |      673.087 |      286.100
 frontend     | /product/{id}                       |      752.567 |       752.567 |      307.197 |      118.877
 frontend     | /cart                               |      582.594 |       582.594 |      270.289 |       92.472
 checkout     | hipstershop.CurrencyService/Convert |      193.229 |       193.229 |      101.888 |       30.177
 frontend     | /setCurrency                        |        7.504 |         7.504 |        0.908 |        0.498
 payment      | grpc.grpc.health.v1.Health/Check    |        1.615 |         1.615 |        0.172 |        0.386
 currency     | grpc.grpc.health.v1.Health/Check    |        1.355 |         1.355 |        0.140 |        0.420If we look at the results we can for example quickly see that there seems to be an issue with the /cart/checkout endpoint. Note that if we just looked at the average we would think performance is good (600 ms). Even if we look at the 95% percentile performance still looks acceptable at 1.2 seconds. However, when we look at the 99% percentile we can see the performance is extremely poor (20 seconds). So somewhere in between 1% and 5% of the requests delivered a very poor user experience. This is definitely something worth investigating further.As we can see, the results of the query are showing the performance of automated health checks. That may be useful in some scenarios but it’s not something our users experience and we may want to filter them out. To do it we would just add an additional condition to the where clause:start_time > NOW() - INTERVAL '1 hour' AND
parent_span_id = 0 AND
span_name NOT LIKE '%health%'Query 3: Identify what services generate more load on other services by operation in the last 30 minutesIn microservice architectures, there are many internal calls between services. One of those microservices could be going through a lot of load. That may be a service that is used by several other services for performing different operations and we would not immediately know what’s causing that.Using the query below we can quickly list all dependencies across services and get an understanding of not only who is calling who and how often, but also what are the specific operations involved in those calls and how long the execution of those calls is taking in aggregate (in seconds). This serves as an additional indicator of load since some types of requests could be much more expensive than others.SELECT
    client_span.service_name AS client_service,
    server_span.service_name AS server_service,
    server_span.span_name AS server_operation,
    count(*) AS number_of_requests,
    ROUND(sum(server_span.duration_ms)::numeric) AS total_exec_time
FROM
    span AS server_span
    JOIN span AS client_span
    ON server_span.parent_span_id = client_span.span_id
WHERE
    client_span.start_time > NOW() - INTERVAL '30 minutes' AND
    client_span.service_name != server_span.service_name
GROUP BY
    client_span.service_name,
    server_span.service_name,
    server_span.span_name
ORDER BY
    server_service,
    server_operation,
    number_of_requests DESC;In this query we are leveraging another powerful capability of SQL: joins. We are joining the span table with itself to identify only the spans that represent a call between two services. This is what the conditionclient_span.service_name != server_span.service_nameaccomplishes.The result would look something like the following:client_service | server_service |                    server_operation                     | number_of_requests | total_exec_time
----------------+----------------+---------------------------------------------------------+--------------------+-----------------
 frontend       | adservice      | hipstershop.AdService/GetAds                            |               2672 |               1
 frontend       | cart           | hipstershop.CartService/AddItem                         |                509 |               5
 checkout       | cart           | hipstershop.CartService/EmptyCart                       |                174 |               2
 frontend       | cart           | hipstershop.CartService/GetCart                         |               3697 |              25
 checkout       | cart           | hipstershop.CartService/GetCart                         |                174 |               2
 frontend       | checkout       | hipstershop.CheckoutService/PlaceOrder                  |                174 |              57
 frontend       | currency       | grpc.hipstershop.CurrencyService/Convert                |               8635 |              14
 checkout       | currency       | grpc.hipstershop.CurrencyService/Convert                |                408 |               1
 frontend       | currency       | grpc.hipstershop.CurrencyService/GetSupportedCurrencies |               3876 |               3
 checkout       | email          | /hipstershop.EmailService/SendOrderConfirmation         |                174 |               0
 checkout       | payment        | grpc.hipstershop.PaymentService/Charge                  |                174 |               0
 frontend       | productcatalog | hipstershop.ProductCatalogService/GetProduct            |              20436 |               1
 checkout       | productcatalog | hipstershop.ProductCatalogService/GetProduct            |                234 |               0
 frontend       | productcatalog | hipstershop.ProductCatalogService/ListProducts          |                501 |               0
 frontend       | recommendation | /hipstershop.RecommendationService/ListRecommendations  |               3374 |              15
 frontend       | shipping       | hipstershop.ShippingService/GetQuote                    |               1027 |               0
 checkout       | shipping       | hipstershop.ShippingService/GetQuote                    |                174 |               0
 checkout       | shipping       | hipstershop.ShippingService/ShipOrder                   |                174 |               0These results show the client service making the request in the first column and the service and operation receiving the request in the second and third columns. We see for example that the GetProduct method of the ProductCatalogService has been called from the frontend service and the checkout service and that the former made many more calls in the last 30 minutes, which is expected and not an issue in this case. But if we saw a much higher percentage of calls to the GetProduct method originating from the checkout service this would be an indicator of something unexpected going on.ConclusionTraces are extremely useful to troubleshoot and understand modern distributed systems. They help us answer questions that are impossible or very hard to answer with just metrics and logs. Adoption of traces has been traditionally slow because trace instrumentation has required a lot of manual effort and existing observability tools have not allowed us to query the data in flexible ways to get all the value traces can provide.This is not true anymore thanks to OpenTelemetry and Promscale.OpenTelemetry is quickly becomingtheinstrumentation standard and it offers libraries that automate (or at the very least simplify) trace instrumentation dramatically reducing the amount of effort required to instrument our services. Additionally, the instrumentation is vendor agnostic and the traces it generates can be sent to any compatible observability backend so we can change backends or use multiple ones.Promscale is the observability backend with full SQL support for querying traces and metrics. With Promscale you can query your data to answer any question you need to answer.The combined power of OpenTelemetry and Promscale help you get more value out of traces and build better software.Get started with PromscaleTo start getting more value out of your traces and metrics with Promscale:Check out our tracing documentationfor more details on how to start collecting traces with OpenTelemetry, Jaeger, and Zipkin and how to visualize them in Jaeger and Grafana.Learn about Promscale installationfor more on how Promscale works with Prometheus, installation instructions, sample PromQL and SQL queries, and more.Check ourGitHub repository. As a reminder, Promscale is open-source and completely free to use. (GitHub ⭐️  welcome and appreciated! 🙏)Promscale is also available on Timescale.Get started now with free 30-day trial (no credit card required).WatchPromscale 101 YouTube playlistfor step-by-step demos and best practices.Whether you’re new to Promscale or an existing community member, we’d love to hear from you!Join TimescaleDB Slack, where you’ll find 7K+ developers and Timescale engineers active in all channels. (The dedicated#promscalechannel has 2.5K+ members, so it’s a great place to connect with like-minded community members, ask questions, get advice, and more).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/what-are-traces-and-how-sql-yes-sql-and-opentelemetry-can-help-us-get-more-value-out-of-traces-to-build-better-software/
2023-06-29T14:24:25.000Z,Savings Unlocked: Why We Switched to a Pay-For-What-You-Store Database Storage Model,"We have great news: storing data in your Timescale database is now cheaper and more flexible! Effective immediately, you will only pay for the storage you actually use in your Timescale services, without pricing gotchas or hidden costs. This new storage experience is simple, transparent, and saves you money—especially when combined with features like compression and data tiering.If you’re not yet a Timescale user, you can try it today for free.Last May, we shared our renewed vision for Timescale: a simpler, faster, and more cost-effective PostgreSQL (or PostgreSQL++ as we call it). Today, we’re excited to announce the first of the many improvements we’re working on to get us closer to this vision: our storage pricing is now usage-based!From now on, you will only be charged for the actual data you store in your Timescale services. You will no longer need to allocate a fixed storage volume or worry about managing your disk size, and if you compress or delete data, you will immediately see a reduction in your bill.Now,you can solely focus on managing your time-series data, not your storage. Wondering what this means for you? Keep reading.Usage-Based Database Storage, Meet TimescaleUntil now, Timescale’s database storage was allocation-based. When our customers created a service, they assigned a disk size from a list of storage plans, with options ranging from 50 GB to 16 TB.Before: our old database storage experience. When creating a service, you had to choose a fixed disk size, which you could later scale up.This allocation-based model served our customers’ needs well, but we knew we could do better. This model did not completely align with our vision of Timescale as an easy-to-use and flexible tool designed to simplify developers' workflows; most importantly, being tied to a particular pre-allocated database storage volume was holding our customers back fromimmediatelysaving money.Many of our best features (likecompression) allow our customers to do more with fewer resources: it was only a logical next step to liberate our services from fixed disks, giving them the flexibility to stop paying for the excess storage they no longer needed.After: our new storage experience. Instead of thinking about disk allocation, you can jump straight to ingesting data, paying only for what you use.Our new database storage model solves this tension. When creating a Timescale service, you no longer need to worry about the best disk size for your use case. Instead, you can start ingesting data with as little as a few MBs.You may be thinking, ""This sounds awesome—but why do I still need to worry about which compute is best for me?"" Our answer: stay tuned. 🔥 We got something cookin'!Is Timescale’s database storage pricing really that simple? Yes!If youdelete some data later, you stop paying for it.If youenable compression, you begin payingway lessfor storage.If youtier data to object storage, your bill will shrink even more.You’ll receive a simplified bill that is easy to understand (based on only two metrics, compute and storage used) and transparent (your consumption is always accessible in the console). And what’s best, you don’t have to worry about scaling up your disk, downtime, or downsizing.Make the Most of Our Usage-Based Database Storage PricingUsage-based database storage pricing is becoming increasingly popular among modern cloud databases because it suits the on-demand nature of cloud architectures.Moreover, we believe this model will be especially powerful for data-intensive time-series applications when paired with Timescale features. These are engineered to let you manage your data effectively without affecting performance. If you can do more with less by paying only for what you use in Timescale, you’ll keep more money in your pocket!Want to learn more? Let us share how you can immediately benefit from our new usage-based database storage experience.Don’t be locked into a diskWe all know the feeling: it’s time to create a new database service, but which disk size to choose? In an allocation-based model, it is hard to downsize a previously allocated disk: if we choose poorly, we may end up paying for a half-empty data storage plan. Similarly, when it’s time to scale, sometimes we only need a bit more storage for our database, yet we’re forced to pay for the next available plan (often much more expensive). In Timescale, this is no longer an issue: you don’t have to think about storage plans when creating a service or while running it.Immediately save money with compressionNative columnar compression was a star feature before, but now it’s the crown jewel. For example, if compression saves you 5x on database storage overall (a common value for our customers), your effective storage cost will now lower 5x too!Our compression is now fully mutable, allowing for DML operations (including deletes and updates), and it also improves performance for many common queries.Make sure to enable it when creating your hypertables.Migrate your database smoothlyMany Timescale customers migrate their PostgreSQL database from Amazon RDS PostgreSQL to Timescale. If this is you, you can now migrate your data progressively and on your own terms without the need to previously allocate disks to fit your newly ingested data or watch out for potentially full disks. Even better, you can immediately compress your data as soon as it gets into Timescale to save money compared to RDS (or tier it to object storage for even more savings).Start smallThis new storage experience makes Timescale more accessible for projects of all shapes and sizes, which makes us very happy! Are you developing a proof of concept (PoC) or playing around with a small prototype with only a few MBs of data? No problem: you can pick our smaller compute capacity and store your time-series data in a high-performance service for a very affordable price. If your project takes off, you already have all you need to scale.Scale seamlesslyWhether you start from scratch or with a multi-TB database, your Timescale service will grow with you. You won’t have to worry about last-minute storage resizing, and your database storage bill won’t suddenly spike due to plan upgrades. You can fit up to 16 TB of datapost-compressionin our regular storage (this isa lotof compressed data!), and if you’re looking for a cheaper way to scale…Tier data (and save even more money)…you can tier your older data to save even more money!By running a simple command on your hypertable, you can automatically tier older data into a low-cost, built-in object storage layer built on Amazon S3. This object store is also usage-based, with no volume limitations (you can tier as much data as you want), and you can keep accessing your data via standard SQL.Take advantage of continuous aggregatesContinuous aggregates, our improved version of PostgreSQL incrementally-updated materialized views, can also help you do more with less, which in a usage-based model meansmore money saved.  Continuous aggregates allow you to createblazing-fast viewsin the millisecond range even when your raw volume is growing heavily.Our customers often use continuous aggregates to power their real-time views and user-facing historical analytics. Continuous aggregates stay around even when underlying raw data has been deleted, allowing you to effectivelydownsampleyour dataset (while paying less for database storage) without affecting performance.Create cheaper replicas (in one click)Not being locked into empty disk space saves you money for your primary service but also your replicas. In Timescale, it only takes one click to createhigh-availability replicas(to protect you even further from potential downtime) andread replicas(to liberate load from your write-heavy primary service). With this new storage experience, your replica will perfectly match the usage in your primary at all times, therefore staying cheaper than if you were replicating (and paying for) a partially empty disk.Don’t worry about pricing plansOur new pricing model is as simple as it can be: $0.001212 per GB-houracross all services and regions, with disk usage metered at 15-minute intervals. Remember that this is what you'll be chargedpost-compression: if your 100 GB turn into 20 GB, your effective price per GB will lower 5x!Many of our customers experience even higher compression rates, so make sure toenable compressionso you don't leave money on the table!And relax: there are no hidden costs!Lastly, you may like surprises—but you surely don’t want them in our bill. Price transparency and simplicity are essential to us. This is why we calculate your Timescale bill based on two metrics: compute and storage.There is no need to think about data transfer, reads and writes, or any other hidden costthat inflates your bill.You also have access to top-rated technical support at no extra cost: we’ll walk alongside you at every step of your journey, helping you get the best out of Timescale from PoC to production and beyond.Get Started TodayAll Timescale services are already benefiting from this new database storage experience. It is the first step toward many great things we’re developing to make working with time series and relational data simpler, faster, and more cost-effective. We can’t wait to show you what’s next!If you’re still not using Timescale,you can create your first service in just two minutes.And if you’d like personalized advice for your use case,reach out to us. We’d love to hear from you!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/savings-unlocked-why-we-switched-to-a-pay-for-what-you-store-database-storage-model/
2022-02-22T15:02:37.000Z,Year of the Tiger: $110 million to build the future of data for developers worldwide,"Timescale just raised $110 million in our Series C, led by Tiger Global alongside all existing investors: Benchmark, New Enterprise Associates, Redpoint Ventures, Icon Ventures, and Two Sigma Ventures. With this funding, Timescale is now valued at over $1 billion, and combined with earlier rounds (2018,2019,2021), has raised over $180 million to fuel its growth. In the past two years, Timescale has seen 7x community growth and 20x revenue growth, with over 500 paying customers and tens of thousands of other organizations using TimescaleDB in the community today. (And yep, the Timescale Tigers raised money from Tiger Global during the Year of the Tiger🐯🐯🐯🦄🚀💥.We don’t believe in coincidences😉.)All data is time-series data.This was our thesis when we launched Timescale 5 years ago.We saw that time-series data was becoming ubiquitous, in part because of major trends like the rise of IoT / machine data, IT observability, and more recently, web3 / crypto applications. But also because we saw businesses in every industry starting to collect time-series data to build exceptional data-driven products and product experiences.This might be measuring the temperature and humidity of soil to help farmers combat climate change. Or tracking streaming metrics to identify popular music and artists. Or analyzing NFT activity in real-time. Or building new gaming experiences. Or observing every action that a user takes in a software application, and the performance of the infrastructure underlying that application, to help resolve support issues and increase customer happiness.As computing continues to get more powerful, and storage gets even cheaper, developers are able to collect data at higher fidelities than before, enabling them to build radically better data-driven product experiences and businesses. Time-series - tracking trends, not static data points - represents data at the highest fidelity.5 years ago we also saw that time-series data is relentless, creating performance and scaling challenges that traditional databases were not built to handle.That’s when we realized that what these new time-series applications needed was a new kind of database, something built for scale and performance, but also something that offered the reliability and versatility only found in relational databases.This thesis also included two other core beliefs: PostgreSQL is the best foundation for software applications; and SQL is, and will continue to be, the universal language for data analysis.That is why we built TimescaleDB, the first-ever relational database for time-series.Welcoming Tiger Global to the teamThere are many successful data platforms today, like Snowflake (for data analysts) and Databricks (for data scientists).Our vision is to build a data platform for developers, anchored around a best-in-class developer experience for PostgreSQL, time-series data, analytics, and data-driven applications.With this new round of financing, our second financing in the last 12 months, we are accelerating towards this vision: investing in product, engineering, R&D; serving our community, customers, and developers worldwide; and growing our amazing team.We’re also using this financing as an opportunity to give back to the larger developer community. For example, while we’ve employed PostgreSQL contributors for a few years now, with this financing we are building an entire team dedicated to upstream PostgreSQL contributions (anyone interested can applyhere).And, with this financing, we are adding another great partner to the team, Tiger Global:""Timescale's laser focus on driving community and consumption around its time-series database is what stood out to us. As time-series data becomes even more pervasive, Timescale is playing a pivotal role in how companies work with data."" said John Curtius, Partner at Tiger Global. ""They're a team that is committed to, and that we believe will, build the next great database company.""They’re joining our existing investors Benchmark, New Enterprise Associates, Redpoint Ventures, Icon Ventures, and Two Sigma Ventures:""We partnered with Timescale over 4 years ago because we saw that all data was fundamentally time-series data, and that Timescale was uniquely poised to alleviate the pain felt by developers. Since then we have seen the team make true on their promise and witnessed their unwavering devotion to serving developers globally,"" said Peter Fenton, General Partner at Benchmark. ""Sitting at the intersection of three transformative trends - time-series, PostgreSQL, and SQL - Timescale is building a foundational database company that is quickly becoming an indispensable part of the software application stack.""The large and rapidly growing TimescaleDB developer communitySource:https://tradingstrategy.ai/blog/building-cryptocurrency-websiteTimescale usage by our customers and community continues to grow exponentially every year, including by companies like Apple, Akamai, Bosch, Cisco WebEx, Comcast, DigitalOcean, Disney, Electronic Arts, GE, IBM, Marvel Studios, Microsoft, Nutanix, NYSE, Pfizer, Samsung, Schneider Electric, Siemens, Tesla, Uber, Walmart, and tens of thousands of others:That’s the quantitative data. But there’s also the qualitative perspective on how developers use and view TimescaleDB today:A cross-section of what developers say about TimescaleDB.Source:CryptoBoole,throrin19,JustJake,wollud1969,apenwarr,zswaff,abrookins,sriramskota,larshmp,jeffbarg,claytonyochum,olithissen.Timescale today: PostgreSQL for time-seriesTimescaleDB is engineered on top of PostgreSQL, which means that TimescaleDB offers everything that PostgreSQL offers plus additional capabilities for time-series workloadsTimescaleDB is the first-ever relational database for time-series.A database purpose-built for time-series data, engineered on top of PostgreSQL (packaged as a PostgreSQL extension), with full SQL support. 100% free, open source (“open core”, to be precise), and petabyte-scale. A database where you can store time-series data alongside ordinary data and get the best of both worlds.We are big fans of PostgreSQL. Even with its long history, today PostgreSQL is still one of the fastest growing databases in terms of usage and community size. Its popularity is due in large part to the hard work and dedication of the PostgreSQL core developer community towards building a reliable and versatile database.One of the many great features of PostgreSQL is that it is designed to beextensible. These “PostgreSQL extensions” add extra functionality without slowing down or adding complexity to core development.While many PostgreSQL extensions only add new functions or datatypes, we’ve leveraged this framework extensively to build a radically better database for time-series workloads, while preserving all the goodness of PostgreSQL. (Notably, this means that TimescaleDB is not a fork but an extension of PostgreSQL - so it stays aligned with core / mainline PostgreSQL.)Some of the groundbreaking capabilities that we’ve added over the past 5 years include:hypertables(the illusion of a single table across all space and time, despite 2D chunking),columnar compression in a row-oriented database(90%+ compression using best-in-class compression algorithms),continuous aggregatesandreal-time aggregation(real-time, incremental materialized views),distributed hypertables that grow to petabyte-scale(horizontally scaling out writes and reads across multiple nodes),hyperfunctions(new SQL functions to simplify working with data in PostgreSQL),function pipelines(functional programming in PostgreSQL using custom operators), and more.And we’ve done this while maintaining all of the goodness of PostgreSQL and the PostgreSQL ecosystem:All of the flexibility and power of SQL (yes, full SQL, not “SQL-like” or “SQLish”)All of the PostgreSQL compatible language and ORM connectors (Python, JavaScript, Ruby, Go, R, Django, Node.js, and so many more)Geospatial support viaPostGIS, and other domain specific capabilities from the breadth of PostgreSQL extensionsAll of the management and administration tools already available for PostgreSQL (e.g., backup / restore, high-availability, physical replication)All of the SQL compatible visualization tools and connectors like Tableau, Looker, Grafana, PowerBI, Retool, Metabase, and moreAs a result, we’ve built not just a better PostgreSQL for time-series, but also a best-in-class product that outperforms other databases likeMongoDB,Cassandra,AWS Timestream,Clickhouse,InfluxDB, and others fortime-series workloads. This is because TimescaleDB, unlike general purpose databases, is purpose-built for time-series; and also because TimescaleDB, unlike other time-series databases,is not just a time-series database but also a relational database, i.e., a PostgreSQL database, all-in-one.In fact, we’ve done a lot more over the past 5 years:Developed a new licensing modelthat allows us to make all of our software free, while being able to build a self-sustaining business, on top of our open-source Apache 2 licensed coreLaunched a fully-managed TimescaleDB cloud serviceon AWS, Azure, and GCPLaunched another core software product, Promscale, as an observability backend powered by TimescaleDB (and then added support to Promscale forcollecting traces via OpenTelemetry)We also built a remote-first culture and globally distributed, diverse team of 100+ amazing individuals across 20+ countries and 6 continents. (And we’re still hiring!)And, thanks to all of these efforts,built a business that has seen 7x community growth and 20x revenue growth in just the last 2 years, with over 500 paying customers and tens of thousands of other organizations using TimescaleDB in our community today.“Timescale has built a brilliant abstraction layer on top of PostgreSQL that lets us treat our gigantic / arbitrarily large data sets like they're in one table. At Messari, we have time-series data coming in by the tens or hundreds of thousands of rows per minute, 24/7/365. Timescale lets us manage this gigantic throughput in what appear at first to be simple, singular SQL tables. Because it's Postgres, there are no surprises in terms of how to query the data later and it's a lot easier to build new functionality on top of our core database layer than with similar products. - Adam Inoue, Software Engineer,Messari“TimescaleDB has absolutely changed my perspective on how relational databases can perform so well in ingesting and querying time series data. We were using a SQL database and decided to give a try with TimescaleDB and it did not disappoint. The community on Slack is extremely helpful. Keep up the good work TimescaleDB!” - Esther Toh, Senior Lead Software Engineer,Bridgestone Asia Pacific“TimescaleDB is one of those rare pieces of technology that is both extremely capable yet easy to learn. The more familiar you become with TimescaleDB, the more you'll see what it is capable of. Not to mention, the TimescaleDB docs are some of the most thorough and well organized docs you'll find for any programming tool.” - Alex Koutmos, Senior Software Engineer,Whoosh.ioTimescale tomorrow: Building a better PostgreSQLWe are grateful for the opportunity to serve this growing community and developers worldwide. We’ve made a lot of progress in the past 5 years, but, of course, we’re just getting started.Every company today is either a software company, becoming a software company, or getting replaced by a software company. Developers are the vanguard of this transformation.What these developers need isn’t just a better time-series database, buta better PostgreSQL for their workloads.We started off building a “PostgreSQL for time-series.” But to our community, we are also: “PostgreSQL for IoT”, “PostgreSQL for web3”, “PostgreSQL for analytics”, “PostgreSQL for observability”, “PostgreSQL for gaming”, “PostgreSQL for events”, and more.Looking ahead, our goal is to keep innovating on top of PostgreSQL and to continue adding breakthrough capabilities that enable more developers to build exceptional data-driven applications.To name a few engineering efforts currently underway and slated for release this year:An enhanced cloud-native PostgreSQL experience, including at scale with cost-effective serverless storage, as well as performance and operational improvements to horizontally scale-out TimescaleDBAll-around performance improvements and improved developer ergonomics for analyzing data in PostgreSQLMore improvements to Promscale, the observabilty backend powered by SQL and TimescaleDBAn enhanced developer console and cloud APIs to better manage and understand your fleet of databasesAnd much, much more.Come join us!To all our users, we thank you for your support and feedback, and for building alongside us. To everyone who is not yet a user, we invite you totry Timescale for free today.Once you are using TimescaleDB, please join theTimescaleDB community in Slackor inour new forumsand ask any questions you may have about time-series data, databases, and more.To learn more about Timescale and how we can help you build better data-driven product experiences and businesses, join us for the nextTimescale Community Day on March 31, 2022.Also, tomorrow February 23rd, my cofounderMikeand I will host a conversation about time-series data and the future of developer data platforms onTwitter Spacesat 11h00 PT/ 14h00 ET / 19h00 GMT.And, for those who share our mission and values, and want to join our fully remote, global team:We’re hiring!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/year-of-the-tiger-110-million-to-build-the-future-of-data-for-developers-worldwide/
2023-11-01T12:15:39.000Z,Database Monitoring and Query Optimization: Introducing Insights on Timescale,"When you create the first version of your app, all your queries are probably fast. The data being managed is not large yet, so everything fits in memory and finishes in milliseconds. Instead of monitoring your database and your queries, you spend your time elsewhere, maybe optimizing some code paths or tweaking your infrastructure to manage any slowdowns.But, as you keep growing, with more data, you might find yourself looking at the database as the culprit. It might not be every query or every time it runs, but now you need to do some digging around database monitoring and Postgres performance, and it’s time-consuming.And that is where Timescale’s Insights comes in. Released today to General Availability in all regions as part of ourCloud Week.Insights allows you to deeply observe the queries that have been running in your database over a given time period, with useful statistics on timing/latency, memory usage, and more. Once you’ve pinpointed a specific query (or queries) for further digging, you can see even more information to help you understand what’s going on, including whether your database performance has been degrading or improving (e.g., because of a newly added index).If you have been usingpg_stat_statementsto optimize andmonitor your queriesto keep your Postgres performance in check, Insights provides a complementary and flexible form of database observability but at an unprecedented level of granularity.Keep reading to learn how Insights works and how you can save precious time and effort optimizing your queries.A new database and query monitoring tool: How does it work?This year, we built a scalable query collection system to collect sanitized statistics on every query running in our cloud service. We store all this data in Timescale (our fully managed cloud solution that makes PostgreSQL faster and fiercer), which can then be used to power the graphs and stats you see in Insights.In a future blog post, we will explore the lessons we learned while doing this, which turned into the largestdogfooding effortwe’ve undertaken. The scale has been massive and is a testament to our product’s (and team’s) capability: over1 trillionnormalized queries (i.e., queries whose parameter values have been replaced by placeholders) have been collected, stored, and analyzed, with over10 billionnew queries ingested every day.In doing so, we've utilized nearly all the features across our offering: continuous aggregates, compression, data tiering, connection pooling, replicas, and yes, even Insights itself, while building this.But, as I said, more on that in a future blog post.Why pg_stat_statements isn’t enoughPostgreSQL comes with some standard tools to provide some of the statistics you can find in Insights. For example, pg_stat_statements is an official extension that can be installed to help see statistics from within your database. In fact,all Timescale services come with it enabled by default.However, there are limits to this approach:It doesn’t store every query you run over longer timespans (i.e., days or weeks), given the overhead associated with its approach.The “system” queries can pollute the view.It is limited to PostgreSQL-level info, so useful stats from extensions are not visible.With our collection system, we move to storing queries and their statistics in a database designed for vast amounts of data. We are able to store query statistics for longer time periods, so you can see how your query’s performance has evolved over time. We filter out queries that are not important to your app, improving the signal-to-noise ratio. And we keep statistics about Timescale feature usage, both to (1) allow you to see how hypertables, continuous aggregates, and so on help you and (2) help us determine how well key features are working.Getting Database Monitoring InsightsThe Insights feature is available in the service view of your Timescale service. The initial view can tell you a lot, but it is also potentially just the starting point of your journey.Database monitoring: Insights' initial view into your query latency, CPU, memory, and storage I/O usageThe first thing you’ll see is a graph that lets you explore the relationship between different system resources—CPU, memory, and disk I/O—and query latency. You can zoom in, e.g., to better understand a spike in disk I/O by seeing which queries were running.Changing the time period via zoom or using the filters at the top will update this graph view with the relevant queries. Your database is likely doing several things at once, so we don’t clutter the graph with every query running in that time frame. If you want to change which queries you’re looking at, scrolling down to the table below lets you do that.How to optimize queries: Check your top 50 queries using InsightsThe table shows you more info about not just the queries in the graph but the top 50 queries (based on your chosen sort) running in the time frame. You can see which queries execute most often, affect the most rows, take the longest time, and are utilizing Timescale features like hypertables and continuous aggregates.Sometimes, that’s enough to help you track down database issues. You’ll see that a query is running more often than it used to, touching more rows than you anticipated, or it’s time to consider making that really large table into ahypertable(which enables the automatic partitioning of your data). But if it’s not, you can go deeper on individual queries.Drilling down on a queryWhether you know a particular query is problematic but aren’t sure why, or you’re still trying to determine if a query is the problem, the drill-down view of Insights provides you with finer-grain metrics to sift through.Database monitoring and query optimization made easy: Drilling down on queries with InsightsInstead of providing aggregates over a given time period, the drill-down view will show you trends across metrics like latency, shared buffer usage, cache utilization, and so on. All the graphs are fully zoomable if you need to pinpoint a tighter time range or if you want to see how the time range you’re looking at compares to a broader picture.This will allow you to quickly identify if you’ve experienced a few outlier or anomalous query runs, or if performance is trending in the wrong direction.Real Use Case Example: Humblytics StoryTo illustrate how you can use Insights to improve your PostgreSQL performance, we’ll share an example by our customerHumblytics.The team at Humblytics is building a privacy-friendly analytics tool designed for Webflow. They use Timescale to trackcustom events, taking the burden off their users so they can directly extract information about their Webflow website interactions.As they sharedin this Twitter thread, the team was scaling quickly and realized something abnormal. After observing that their CPU load was close to 100 percent, they assumed that since they were adding new customers to their platform and their load was increasing, perhaps it was time to upscale their instance—so they added more CPU.However, this didn’t seem to help. Their instance quickly maxed out again.The Hymblytics team jumped into Insights (available in beta at the time under the label “Query Stats”) and immediately saw what was wrong. The issue wasn’t insufficient CPU: it was a matter of fixing a problematic query, anUPDATEquery touching more records than it was supposed to. It was an easy fix, and everything went back to normal quickly after identifying the problem.As they added:  ""Without Timescale, we would've been at risk of churning customers. Instead, everything was gravy 😊""0:00/0:521×A special thank you to Humblytics for sharing their story ❤️Getting Insights on InsightsAs mentioned before, Insights is built on our most ambitious dogfooding effort to date. There will be a follow-up post to dive more into that, but one tidbit illustrates Insights’ usefulness.The system storing all the data backing Insights is a standard Timescale service on our cloud. That is, it has no special infrastructure setup or unique features that are not currently available (or, at worst, will soon be available) to all our customers. So when we were testing out Insights and tracking down performance issues internally, we turned to Insights to help us track down which queries were slow or which suddenly performed worse.This was a powerful tool. Wediscovered a few TimescaleDB performance bugsthat might otherwise have persisted, and being able to visually observe performance over time as we changed key database parameters was immensely useful. We're excited to have Insights available to all our customers.Our goal: the next time you are concerned with database performance or are debugging an underperforming query, Insights can help you reach a solution faster and easier.To try out Insights and unlock a new, insightful understanding of your queries and performance,sign up for Timescale. It’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/database-monitoring-and-query-optimization-introducing-insights-on-timescale/
2021-10-19T13:33:44.000Z,Function Pipelines: Building Functional Programming Into PostgreSQL Using Custom Operators,"Today, we are announcingfunction pipelines, a new capability that introduces functional programming concepts inside PostgreSQL (and SQL) using custom operators.Function pipelinesradically improve the developer ergonomics of analyzing data in PostgreSQL and SQL, by applying principles fromfunctional programmingand popular tools likePython’s PandasandPromQL.At Timescale our mission is to serve developers worldwide, and enable them to build exceptional data-driven products that measure everything that matters: e.g., software applications, industrial equipment, financial markets, blockchain activity, user actions, consumer behavior, machine learning models, climate change, and more.We believe SQL is the best language for data analysis. We’ve championed the benefits of SQL for several years, even back when many were abandoning the language for custom domain-specific languages. And we were right - SQL has resurged and become the universal language for data analysis, and now many NoSQL databases are adding SQL interfaces to keep up.But SQL is not perfect, and at times can get quite unwieldy. For example,SELECT device_id, 
	sum(abs_delta) as volatility
FROM (
	SELECT device_id, 
		abs(val - lag(val) OVER (PARTITION BY device_id ORDER BY ts))
        	as abs_delta 
	FROM measurements
	WHERE ts >= now() - '1 day'::interval) calc_delta
GROUP BY device_id;Pop quiz: What does this query do?Even if you are a SQL expert, queries like this can be quite difficult to read - and even harder to express. Complex data analysis in SQL can be hard.Function pipelineslet you express that same query like this:SELECT device_id, 
	timevector(ts, val) -> sort() -> delta() -> abs() -> sum() 
    		as volatility
FROM measurements
WHERE ts >= now() - '1 day'::interval
GROUP BY device_id;Now it is much clearer what this query is doing. It:Gets the last day’s data from the measurements table, grouped bydevice_idSorts the data by the time columnCalculates the delta (or change) between valuesTakes the absolute value of the deltaAnd then takes the sum of the result of the previous stepsFunction pipelines improve your own coding productivity, while also making your SQL code easier for others to comprehend and maintain.Inspired by functional programming languages, function pipelines enable you to analyze data by composing multiple functions, leading to a simpler, cleaner way of expressing complex logic in PostgreSQL.And the best part: we built function pipelines in a way that is fully PostgreSQL compliant - we did not change any SQL syntax - meaning that any tool that speaks PostgreSQL will be able to support data analysis using function pipelines.How did we build this? By taking advantage of the incredible extensibility of PostgreSQL, in particular:custom types,custom operators, andcustom functions.In our previous example, you can see the key elements of function pipelines:Custom data types: in this case, thetimevector, which is a set of(time, value)pairs.Custom operator:->, used tocomposeandapplyfunction pipeline elements to the data that comes in.And finally,custom functions; calledpipeline elements.Pipeline elements can transform and analyzetimevectors (or other data types) in a function pipeline. For this initial release, we’ve built60 custom functions!(Full list here).We’ll go into more detail on function pipelines in the rest of this post, but if you just want to get started as soon as possible,the easiest way to try function pipelines is through a fully managed Timescale Cloud service.Try it for free(no credit card required) for 30 days.Function pipelines are pre-loaded on each new database service on Timescale Cloud, available immediately - so after you’ve created a new service, you’re all set to use them!If you prefer to manage your own database instances,you can install thetimescaledb_toolkitinto your existing PostgreSQL installation, completely for free.We’ve been working on this capability for a long time, but in line with our belief of “move fast but don’t break things”, we’re initially releasing function pipelines as anexperimental feature- and we would absolutely love toget your feedback. You canopen an issueor join adiscussion threadin GitHub (And, if you like what you see, GitHub ⭐ are always welcome and appreciated too!).We’d also like to take this opportunity to give a huge shoutout topgx, the Rust-based framework for building PostgreSQL extensions- it handles a lot of the heavy lifting for this project. We have over 600 custom types, operators, and functions in thetimescaledb_toolkitextension at this point; managing this withoutpgx(and the ease of use that comes from working with Rust) would be a real bear of a job.Function pipelines: why are they useful?It’s October. In the Northern hemisphere (where most of Team Timescale sits, including your authors), it is starting to get cold.Now imagine a restaurant in New York City whose owners care about their customers and their customers’ comfort. And you are working on an IoT product designed to help small businesses like these owners minimize their heating bill while maximizing their customers happiness. So you install two thermometers, one at the front measuring the temperature right by the door, and another at the back of the restaurant.Now, as many of you may know (if you’ve ever had to sit by the door of a restaurant in the fall or winter), when someone enters, the temperature drops - and once the door is closed, the temperature warms back up. The temperature at the back of the restaurant will vary much less than at the front, right by the door. And both of them will drop slowly down to a lower set point during non-business hours and warm back up sometime before business hours based on the setpoints on our thermostat. So overall we’ll end up with a graph that looks something like this:A graph of the temperature at the front (near the door) and back. The back is much steadier, while the front is more volatile. Graph is for illustrative purposes only, data is fabricated. No restaurants or restaurant patrons were harmed in the making of this post.As we can see, the temperature by the front door varies much more than at the back of the restaurant. Another way to say this is the temperature by the front door is morevolatile. Now, the owners of this restaurant want to measure this because frequent temperature changes means uncomfortable customers.In order to measure volatility, we could first subtract each point from the point before to calculate a delta. If we add this up directly, large positive and negative deltas will cancel out. But, we only care about the magnitude of the delta, not its sign - so what we really should do is take the absolute value of the delta, and then take the total sum of the previous steps.We now have a metric that might help us measure customer comfort, and also the efficacy of different weatherproofing methods (for example, adding one of those little vestibules that acts as a windbreak).To track this, we collect measurements from our thermometers and store them in a table:CREATE TABLE measurements(
	device_id BIGINT,
	ts TIMESTAMPTZ,
	val DOUBLE PRECISION
);Thedevice_ididentifies the thermostat,tsthe time of reading andvalthe temperature.Using the data in our measurements table, let’s look at how we calculate volatility using function pipelines.Note: because all of the function pipeline features are still experimental, they exist in thetoolkit_experimentalschema. Before running any of the SQL code in this post you will need to set yoursearch_pathto include the experimental schema as we do in the example below, we won’t repeat this throughout the post so as not to distract.set search_path to toolkit_experimental, public; --still experimental, so do this to make it easier to read

SELECT device_id, 
	timevector(ts, val) -> sort() -> delta() -> abs() -> sum() 
    	as volatility
FROM measurements
WHERE ts >= now()-'1 day'::interval
GROUP BY device_id;And now we have the same query that we used as our example in the introduction.In this query, the function pipelinetimevector(ts, val) -> sort() -> delta() -> abs() -> sum()succinctly expresses the following operations:Createtimevectors (more detail on this later) out of thetsandvalcolumnsSort eachtimevectorby the time columnCalculate the delta (or change) between each pair in thetimevectorby subtracting the previousvalfrom the currentTake the absolute value of the deltaTake the sum of the result from the previous stepsTheFROM,WHEREandGROUP BYclauses do the rest of the work telling us:We’re getting dataFROMthemeasurementstableWHEREthe ts, or timestamp column, contains values over the last dayShowing one pipeline output perdevice_id(the GROUP BY column)As we noted before, if you were to do this same calculation using SQL and PostgreSQL functionality, your query would look like this:SELECT device_id, 
sum(abs_delta) as volatility
FROM (
	SELECT 
		abs(val - lag(val) OVER (PARTITION BY device_id ORDER BY ts) ) 
        	as abs_delta 
	FROM measurements
	WHERE ts >= now() - '1 day'::interval) calc_delta
GROUP BY device_id;This does the same 5 steps as the above, but is much harder to understand, because we have to use awindow functionand aggregate the results - but also, because aggregates are performed before window functions, we need to actually execute the window function in a subquery.As we can see, function pipelines make it significantly easier to comprehend the overall analysis of our data. There’s no need to completely understand what’s going on in these functions just yet, but for now it’s enough to understand that we’ve essentially implemented a small functional programming language inside of PostgreSQL. You can still use all of the normal, expressive SQL you’ve come to know and love. Function pipelines just add new tools to your SQL toolbox that make it easier to work with time-series data.Some avid SQL users might find the syntax a bit foreign at first, but for many people who work in other programming languages, especially using tools likePython’s Pandas Package, this type ofsuccessive operation on data setswill feel natural.And again, this is still fully PostgreSQL compliant: We introduce no changes to the parser or anything that should break compatibility with PostgreSQL drivers.How we built function pipelines without forking PostgreSQLWe built function pipelines- without modifying theparseror anything that would require a fork of PostgreSQL- by taking advantage of three of the many ways that PostgreSQL enables extensibility:custom types,custom functions, andcustom operators.Custom data types, starting with thetimevector, which is a set of(time, value)pairsAcustom operator:->, which is used tocomposeandapplyfunction pipeline elements to the data that comes in.Custom functions, calledpipeline elements, which can transform and analyzetimevectors (or other data types) in a function pipeline (with 60 functions in this initial release)We believe that new idioms like these are exactly what PostgreSQL wasmeant to enable. That’s why it has supported custom types, functions and operators from its earliest days. (And is one of the many reasons why we love PostgreSQL.)A custom data type: thetimevectorAtimevectoris a collection of(time, value)pairs. As of now, the times must beTIMESTAMPTZs and the values must beDOUBLE PRECISIONnumbers. (But this may change in the future as we continue to develop this data type. If you have ideas/input, pleasefile feature requests on GitHubexplaining what you’d like!)You can think of thetimevectoras something like this:A depiction of atimevector.One of the first questions you might ask is: how does atimevectorrelate to time-series data? (If you want to know more about time-series data, we havea great blog post on that).Let’s consider our example from above, where we were talking about a restaurant that was measuring temperatures, and we had ameasurementstable like so:CREATE TABLE measurements(
	device_id BIGINT,
	ts TIMESTAMPTZ,
	val DOUBLE PRECISION
);In this example, we can think of a single time-series dataset as all historical and future time and temperature measurements from a device.Given this definition, we can think of atimevectoras afinite subset of a time-series dataset. The larger time-series dataset may extend back into the past and it may extend into the future, but thetimevectoris bounded.Atimevectoris a finite subset of a time-series and contains all the(time, value)pairs in some region of the time-series.In order to construct atimevectorfrom the data gathered from a thermometer, we use a custom aggregate and pass in the columns we want to become our(time, value)pairs. We can use theWHEREclause to define the extent of thetimevector(i.e., the limits of this subset), and theGROUP BYclause to provide identifying information about the time-series that’s represented.Building on our example, this is how we construct atimevectorfor each thermometer in our dataset:SELECT device_id, 
	timevector(ts, val)
FROM measurements
WHERE ts >= now() - '1 day'::interval
GROUP BY device_id;But atimevectordoesn't provide much value by itself. So now, let’s also consider some complex calculations that we can apply to thetimevector, starting with a custom operator used to apply these functionsA custom operator:->In function pipelines, the->operator is used to apply and compose multiple functions, in an easy to write and read format.Fundamentally,->means: “apply the operation on the right to the inputs on the left”, or, more simply “do the next thing”.We created a general-purpose operator for this because we think that too many operators meaning different things can get very confusing and difficult to read.One thing that you’ll notice about the pipeline elements is that the arguments are in an unusual place in a statement like:SELECT device_id, 
 	timevector(ts, val) -> sort() -> delta() -> abs() -> sum() 
    	as volatility
FROM measurements
WHERE ts >= now() - '1 day'::interval
GROUP BY device_id;Itappears(from the semantics) that thetimevector(ts, val)is an argument tosort(), the resultingtimevectoris an argument todelta()and so on.The thing is thatsort()(and the others) are regular function calls; they can’t see anything outside of their parentheses and don’t know about anything to their left in the statement; so we need a way to get thetimevectorinto thesort()(and the rest of the pipeline).The way we solved this is by taking advantage of one of the same fundamental computing insights that functional programming languages use:code and data are really the same thing.Each of our functions returns a special type that describes the function and its arguments. We call these typespipeline elements(more later).The->operator then performs one of two different types of actions depending on the types on its right and left sides.  It can either:Applya pipeline element to the left hand argument - perform the function described by the pipeline element on the incoming data type directly.Composepipeline elements into a combined element that can be applied at some point in the future (this is an optimization that allows us to apply multiple elements in a “nested” manner so that we don’t perform multiple unnecessary passes).The operator determines the action to perform based on its left and right arguments.Let’s look at ourtimevectorfrom before:timevector(ts, val) -> sort() -> delta() -> abs() -> sum(). If you remember from before, I noted that this function pipeline performs the following steps:Createtimevectors out of thetsandvalcolumnsSort it by the time columnCalculate the delta (or change) between each pair in thetimevectorby subtracting the previousvalfrom the currentTake the absolute value of the deltaTake the sum of the result from the previous stepsAnd logically, at each step, we can think of thetimevectorbeing materialized and passed to the next step in the pipeline.However, while this will produce a correct result, it’s not the most efficient way to compute this. Instead, it would be more efficient to compute as much as possible in a single pass over the data.In order to do this, we allow not only theapplyoperation, but also thecomposeoperation. Once we’ve composed a pipeline into a logically equivalent higher order pipeline with all of the elements we can choose the most efficient way to execute it internally. (Importantly, even if we have to perform each step sequentially, we don’t need tomaterializeit and pass it between each step in the pipeline so it has significantly less overhead even without other optimization).Custom functions: pipeline elementsNow let’s discuss the third, and final, key piece that makes up function pipelines: custom functions, or as we call them,pipeline elements.We have implemented over 60 individual pipeline elements, which fall into 4 categories (with a few subcategories):timevectortransformsThese elements take in atimevectorand produce atimevector. They are the easiest to compose, as they produce the same type.Example pipeline:SELECT device_id, 
	timevector(ts, val) 
    	-> sort() 
        -> delta() 
        -> map($$ ($value^3 + $value^2 + $value * 2) $$) 
        -> lttb(100) 
FROM measurementsOrganized by sub-category:Unary mathematicalSimple mathematical functions applied to the value in each point in atimevector. (Docs link)ElementDescriptionabs()Computes the absolute value of each valuecbrt()Computes the cube root of each valueceil()Computes the first integer greater than or equal to each valuefloor()Computes the first integer less than or equal to each valueln()Computes the natural logarithm of each valuelog10()Computes the base 10 logarithm of each valueround()Computes the closest integer to each valuesign()Computes +/-1 for each positive/negative valuesqrt()Computes the square root for each valuetrunc()Computes only the integer portion of each valueBinary mathematicalSimple mathematical functions with a scalar input applied to the value in each point in atimevector. (Docs link)ElementDescriptionadd(N)Computes each value plusNdiv(N)Computes each value divided byNlogn(N)Computes the logarithm baseNof each valuemod(N)Computes the remainder when each number is divided byNmul(N)Computes each value multiplied byNpower(N)Computes each value taken to theNpowersub(N)Computes each value lessNCompound transformsTransforms involving multiple points inside of atimevector. (Docs link)ElementDescriptiondelta()Subtracts each value from the previous`fill_to(interval, fill_method)Fills gaps larger thanintervalwith points atintervalfrom the previous usingfill_methodlttb(resolution)Downsamples atimevectorusing the largest triangle three buckets algorithm at `resolution, requires sorted input.sort()Sorts thetimevectorby thetimecolumn ascendingLambda ElementsThese elements use lambda expressions, which allows the user to write small functions to be evaluated over each point in atimevector.Lambda expressions can return aDOUBLE PRECISIONvalue like$$ $value^2 + $value + 3 $$. They can return aBOOLlike$$ $time > ‘2020-01-01’t $$.  They can also return a(time, value)pair like$$ ($time + ‘1 day’i, sin($value) * 4)$$. (Docs link)You can apply them using the elements below:ElementDescriptionfilter(lambda (bool) )Removes points from thetimevectorwhere the lambda expression evaluates tofalsemap(lambda (value) )Applies the lambda expression to all the values in thetimevectormap(lambda (time, value) )Applies the lambda expression to all the times and values in thetimevectortimevectorfinalizersThese elements end thetimevectorportion of a pipeline, they can either help with output or  produce an aggregate over the entiretimevector. They are an optimization barrier to composition as they (usually) produce types other thantimevector.Example pipelines:SELECT device_id, 
	timevector(ts, val) -> sort() -> delta() -> unnest()
FROM measurementsSELECT device_id, 
	timevector(ts, val) -> sort() -> delta() -> time_weight()
FROM measurementsFinalizer pipeline elements organized by sub-category:timevectoroutputThese elements help with output, and can produce a set of(time, value)pairs or a Note: this is an area where we’d love further feedback, are there particular data formats that would be especially useful for, say graphing that we can add?File an issue in our GitHub!ElementDescriptionunnest( )Produces a set of(time, value)pairs. You can wrap and expand as a composite type to produce separate columns(pipe -> unnest()).*materialize()Materializes atimevectorto pass to an application or other operation directly, blocks any optimizations that would materialize it lazily.timevectoraggregatesAggregate all the points in atimevectorto produce a single value as a result. (Docs link)ElementDescriptionaverage()Computes the average of the values in thetimevectorcounter_agg()Computes thecounter_aggaggregate over the times and values in thetimevectorstats_agg()Computes a range of statistical aggregates and returns a1DStatsAggover the values in thetimevectorsum()Computes the sum of the values in thetimevectornum_vals()Counts the points in thetimevectorAggregate accessors and mutatorsThese function pipeline elements act like the accessors that I described in our previouspost on aggregates. You can use them to get a value from the aggregate part of a function pipeline like so:SELECT device_id, 
	timevector(ts, val) -> sort() -> delta() -> stats_agg() -> variance() 
FROM measurementsBut these don’tjustwork ontimevectors - they also work on a normally produced aggregate as well.When used instead of normal function accessors and mutators they can make the syntax more clear by getting rid of nested functions like:SELECT approx_percentile(0.5, percentile_agg(val)) 
FROM measurementsInstead, we can use the arrow accessor to convey the same thing:SELECT percentile_agg(val) -> approx_percentile(0.5) 
FROM measurementsBy aggregate family:Counter AggregatesCounter aggregatesdeal with resetting counters, (and were stabilized in our 1.3 release this week!). Counters are a common type of metric in the application performance monitoring and metrics world. All values have resets accounted for. These elements must have aCounterSummaryto their left when used in a pipeline, from acounter_agg()aggregate or pipeline element. (Docs link)ElementDescriptioncounter_zero_time()The time at which the counter value is predicted to have been zero based on the least squares fit of the points input to theCounterSummary(x intercept)corr()The correlation coefficient of the least squares fit line of the adjusted counter value.delta()Computes the last - first value of the counterextapolated_delta(method)Computes the delta extrapolated using the provided method to bounds of range. Bounds must have been provided in the aggregate or awith_boundscallidelta_left()/idelta_right()Computes the instantaneous difference between the second and first points (left) or last and next-to-last points (right)intercept()The y-intercept of the least squares fit line of the adjusted counter value.irate_left()/irate_right()Computes the instantaneous rate of change between the second and first points (left) or last and next-to-last points (right)num_changes()Number of times the counter changed values.num_elements()Number of items - any with the exact same time will have been counted only once.num_changes()Number of times the counter reset.slope()The slope of the least squares fit line of the adjusted counter value.with_bounds(range)Applies bounds using therange(aTSTZRANGE) to theCounterSummaryif they weren’t provided in the aggregation stepPercentile ApproximationThese aggregate accessors deal withpercentile approximation. For now we’ve only implemented them forpercentile_agganduddsketchbased aggregates. We have not yet implemented them fortdigest. (Docs link)ElementDescriptionapprox_percentile(p)The approximate value at percentilepapprox_percentile_rank(v)The approximate percentile a valuevwould fall inerror()The maximum relative error guaranteed by the approximationmean()The exact average of the input values.num_vals()The number of input valuesStatistical aggregatesThese aggregate accessors add support for common statistical aggregates  (and were stabilized in our 1.3 release this week!). These allow you to compute androllup()common statistical aggregates likeaverage,stddevand more advanced ones likeskewnessas well as 2-dimensional aggregates likeslopeandcovariance.  Because there are both 1D and 2D versions of these, the accessors can have multiple forms, for instance,average()calculates the average on a 1D aggregate whileaverage_y()&average_x()do so on each dimension of a 2D aggregate. (Docs link)ElementDescriptionaverage() / average_y() / average_x()The average of the values.corr()The correlation coefficient of the least squares fit line.covariance(method)The covariance of the values using eitherpopulationorsamplemethod.determination_coeff()The determination coefficient (aka R squared)  of the values.kurtosis(method) / kurtosis_y(method) / kurtosis_x(method)The kurtosis (4th moment) of the values using eitherpopulationorsamplemethod.intercept()The intercept of the least squares fit line.num_vals()The number of (non-null) values seen.sum() / sum_x() / sum_y()The sum of the values seen.skewness(method) / skewness_y(method) / skewness_x(method)The skewness (3rd moment) of the values using eitherpopulationorsamplemethod.slope()The slope of the least squares fit line.stddev(method) / stddev_y(method) / stddev_x(method)The standard deviation of the values using eitherpopulationorsamplemethod.variance(method) / variance_y(method) / variance_x(method)The variance of the values using eitherpopulationorsamplemethod.x_intercept()The x intercept of the least squares fit line.Time Weighted Averages(More info)Theaverage()accessor may be called on the output of atime_weight()like so:SELECT time_weight('Linear', ts, val) -> average()  FROM measurements;Approximate Count Distinct (Hyperloglog)This is an approximation for distinct counts that was stabilized in our 1.3 release!(Docs link) Thedistinct_count()accessor may be called on the output of ahyperloglog()like so:SELECT hyperloglog(device_id) -> distinct_count() FROM measurements;Next stepsWe hope this post helped you understand how function pipelines leverage PostgreSQL extensibility to offer functional programming concepts in a way that is fully PostgreSQL compliant. And how function pipelines can improve the ergonomics of your code making it easier to write, read, and maintain.You can try function pipelines todaywith a fully-managed Timescale Cloud service (no credit card required, free for 30 days). Function pipelines are available now on every new database service on Timescale Cloud, so after you’ve created a new service, you’re all set to use them!If you prefer to manage your own database instances, you candownload and install the timescaledb_toolkit extensionon GitHub for free, after which you’ll be able to use function pipelines.We love building in public. You can view ourupcoming roadmap on GitHubfor a list of proposed features, as well as features we’re currently implementing and those that are available to use today. We also welcome feedback from the community (it helps us prioritize the features users really want). To contribute feedback, comment on anopen issueor in adiscussion threadin GitHub.And if you want to hear more about function pipelines or meet some of the folks who helped build it, be sure to join us for our firstTimescale Community Dayon October 28, 2021!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/function-pipelines-building-functional-programming-into-postgresql-using-custom-operators/
2022-02-23T14:23:08.000Z,Increase Your Storage Savings With TimescaleDB 2.6: Introducing Compression for Continuous Aggregates,"Yesterday, we announced thatwe raised $110 million in our Series C. 🐯🦄🚀 Today, we keep celebrating the big news Timescale-style: with an #AlwaysBeLaunching spirit! We're excited to announce the release of TimescaleDB 2.6, a version that comes with new features highly requested by our community - most notablycompression for continuous aggregates. 🔥 TimescaleDB 2.6 also includes theexperimentalsupport fortimezones in continuous aggregates.We love building in public. We are firm believers in the value of user feedback, as there’s no better way to improve your product than hearing from those who use it daily. We read all the thoughts and comments you share in our Community Slack, and we pay close attention toyour feature requests in GitHub. When features get upvoted, it helps us prioritize what to work on next.With today’s release, we are proud to bring you a top requested feature:the support for compression in continuous aggregates. Originally, we envisioned that compression would mostly be necessary over raw data, as continuous aggregates by themselves help downsample datasets considerably. But TimescaleDB users operate at such a scale thatthey requested compression also for their continuous aggregatesto save even more disk space.Theissuethat originated the support for compression in continuous aggregatesContinuous aggregatesspeed up aggregate queries over large volumes. You can think of them as a more powerful version of PostgreSQL materialized views, as they allow you to materialize your data while your view getsautomatically and incrementally refreshed in the background. But continuous aggregates are also very useful for something else: downsampling. Indeed, another property of continuous aggregates is that you can keep them around even when the data from the underlying hypertable has been dropped. This allows you to reduce the granularity of your data once it reaches a certain age, liberating space while still enabling long-term analytics.The ability to compress continuous aggregates takes this one step further. Starting with TimescaleDB 2.6, you can apply TimescaleDB’snative columnar compressionto your continuous aggregates, freeing even more disk space. And by combiningcompression policies(which automatically compress data after a certain period of time) withdata retention policies, you can automatically set up a downsampling strategy for your older data.Figure describing the downsampling process through continuous aggregates and compression. Notice the relationship between the refresh policy, compression policy, and data retention policy.TimescaleDB 2.6 also comes with another highly requested feature by the community:you're now able to create continuous aggregates with monthly buckets and/or timezonesusingtime_bucket_ng. (Note that this is anexperimentalfeature.)time_bucket_ngis the “new generation” of ourtime_buckethyperfunction, used for bucketing and analyzing data for arbitrary time intervals in SQL. You can think oftime_bucketas a more powerful version of thePostgreSQLdate_truncfunction, allowing for arbitrary time intervals rather than the standard day, minute, hour provided bydate_trunc. Buttime_bucketdoesn’t yet support buckets by months, years, or timezones;time_bucket_ngexpands the capabilities oftime_bucketby including these features in our experimental schema. (We first introduced this featurethanks to an issue from the community.)We’re eager to hear how this works for you. Our main goal with experimental features is to get as much input as possible - please, if you see something that needs improvements,tell us in GitHub!Theissuethat originated the support for timezones in continuous aggregates.​​TimescaleDB 2.6 is available today. If you are already a TimescaleDB user,check out our docs for instructions on how to upgrade. If you are using Timescale Cloud, upgrades are automatic, and no further action is required from your side (you can also start a free 30-day trial, no credit card required).If you are new to Timescale and you want to learn more about continuous aggregates, compression, andtime_bucket_ng, keep reading!Once you’re using TimescaleDB, join our community. You can ask us questions in ourCommunity Slackor in our brand newCommunity Forum, a better home for long-form discussions. We’ll be more than happy to solve any doubts you may have about compression and continuous aggregates, TimescaleDB 2.6, or any other topic. And if you share our mission of helping developers worldwide, we arehiring broadly across many roles!Before moving on, a huge thank you to the team of engineers (and the entire team of reviewers and testers) that made this release possible. This includes the community members who helped us prioritize, develop, and test these features. As we’ve said before, none of this would be possible without your requests, your upvotes, and your feedback.Please, keep ‘em coming!The power of continuous aggregatesContinuous aggregates in TimescaleDBare a more powerful version ofPostgreSQL materialized views. For example, continuous aggregates are incrementally updated with a built-in refresh policy, so they stay up-to-date as new data is added. When querying a continuous aggregate,the query engine will combine the data that is already pre-computed in the materialized view with the newest raw data in the underlying hypertable. In other words, you will always see up-to-date results.TimescaleDB uses internal invalidation records to determine which data has been changed or added in the underlying hypertable since the last refresh; when you refresh your continuous aggregate,only the new or updated data is computed.This means that TimescaleDB doesn’t need to look at your whole table every time you do a refresh, saving tons of computation resources, speeding up the incremental maintenance of continuous aggregate, and allowing you to get faster results in your aggregate queries.This also implies that you can also use continuous aggregates for downsampling. Differently than with materialized views, you are able to retain the continuous aggregate even after the original raw data has been dropped. To reduce the granularity of your time-series data, you can simply define a continuous aggregate and delete your original data once it gets old - we will be demonstrating this process later in this blog post.Compression in TimescaleDBTogether with continuous aggregates,compressionis another oft-used feature of TimescaleDB. By defining acompression policy, TimescaleDB allows you to compress all data stored in a specific hypertable that is older than a specific period of time - let’s say, 7 days old. In this example, you would keep your last week’s worth of data uncompressed, which would allow you to write data into your database at very high rates, giving you optimal query performance for your shallow-and-wide queries as well. Once your data gets old enough (after 7 days), the compression policy would kick in, automatically compressing your data.TimescaleDB is able to give you high compression rates by deployingbest-in-class compression algorithmsalong with anovel hybrid row/columnar storage design. Once a chunk inside your hypertable becomes old enough, TimescaleDB compresses it by storing the columns into an array. In other words: your more recent chunks (in the previous example, all chunks newer than 7 days) will be stored in TimescaleDB as relational, row-based partitions. Once compressed, your chunks will be a columnar store.An example chunk before compression vs after compression. After compression, the data previously stored in multiple rows is now stored as a single row, with the columns being stored as an array.✨ If you want to dive deeper into how compression works in TimescaleDB, check out the following videos:Compression 101 (part 1): learn all the fundamentals of compression and compression policies in TimescaleDB.Compression deep dive (part 2): expand your compression knowledge by getting familiar with more advanced functionality, like how to adequately use and configureSEGMENT BYcolumns.How to use continuous aggregates with compression for downsamplingAs we introduced earlier, continuous aggregates can help you reduce the granularity of your dataset, with the ultimate goal of saving some disk space.We explained previously how, in TimescaleDB, you are able to retain the continuous aggregate even after the original raw data has been dropped. So in order to considerably reduce your data size (and thus your storage costs) automatically and without losing the ability to do long term analytics on your older data, you can:Create a continuous aggregatecapturing the information that you most likely will like to see in your historical analytic queries.Define a refresh policy for your continuous aggregateso your continuous aggregate can stay up to date, periodically materializing your newest data.Enable compressionin the continuous aggregate.Add a compression policyto compress your chunks automatically once they’re older than a specific period of time.Create a data retention policyto automatically drop the raw data in the original hypertable once it gets older than a specific period of time.In this section, we’ll walk you through an example, using financial data fromAlpha Vantage. (If you want to also load this dataset into your TimescaleDB instance, we have astep-by-step guide published in our docs.)We will be using the following schema:CREATE TABLE public.stocks_intraday (
	""time"" timestamptz NOT NULL,
	symbol text NULL,
	price_open float8 NULL,
	price_high float8 NULL,
	price_low float8 NULL,
	price_close float8 NULL,
	trading_volume int4 null
	);
    
SELECT create_hypertable ('public.stocks_intraday', 'time');Create a continuous aggregate and set up continuous aggregate policyTo start, let’screate a continuous aggregateusing TimescaleDB’stime_bucket()function. We will record averages for the price at open, high, low, and close for each company symbol over a given day.We also will set up acontinuous aggregate policy, which will update and refresh data from the last four days. This policy will run once every hour:CREATE MATERIALIZED VIEW stock_intraday_daily
WITH (timescaledb.continuous, timescaledb.materialized_only = true) AS
SELECT 
time_bucket( interval '1 day', ""time"") AS bucket,
AVG(price_high) AS high,
AVG(price_open) AS open,
AVG(price_close) AS close,
AVG(price_low) AS low,
symbol 
FROM stocks_intraday si 
GROUP BY bucket, symbol;Enable and set up compression on your continuous aggregateNow that we have defined the continuous aggregate together with a refresh policy for it, we canenable compressionon this continuous aggregate, also setting up ourcompression policy. This compression policy will automatically compress all chunks older than 7 days:ALTER MATERIALIZED VIEW stock_intraday_daily SET (timescaledb.compress = true);

-- Set up compression policy
SELECT add_compression_policy('stock_intraday_daily', INTERVAL '7 days');Lastly, it is important to notice thatupdating data within a compressed chunk is not supported yet in TimescaleDB. This is relevant for correctly configuring compression policies in continuous aggregates: since refresh policies require chunks to be updated, we have to make sure that our recent chunks remain uncompressed. I.e., make sure you define your time intervals so that your continuous aggregate gets refreshed at a later date than when your compression policy is set.How compression affects storageSo we officially have a compressed continuous aggregate, but you may be wondering: how much of a difference does compression make on this continuous aggregate?In order to find out, let’s check out the stats on our compressed continuous aggregate. To do that, we first need to find the internal name for the materialized hypertable. We can do it by looking at our compressed tables:SELECT * FROM timescaledb_information.compression_settings;Then, we can use that to run thehypertable_compression_stats()command.(Note: The name of your materialized hypertable will most likely be different than the one shown below.)SELECT * FROM hypertable_compression_stats('_timescaledb_internal._materialized_hypertable_3');

Results:
—---------------------------------|--------
total_chunks                   | 7
number_compressed_chunks       | 6
before_compression_table_bytes | 7471104
before_compression_index_bytes | 2031616
before_compression_toast_bytes | 49152
before_compression_total_bytes | 9551872
after_compression_table_bytes  | 401408
after_compression_index_bytes  | 98304
after_compression_toast_bytes  | 3178496
after_compression_total_bytes  | 3678208
Node_nameFor this continuous aggregate, we got a compression rate of over 61%:Size of the continuous aggregate before and after compression (9.6 MB vs 3.7 MB)A 61% compression rate would imply a very nice boost in your storage savings. However, if you’re used to compression in TimescaleDB, you may be wondering: whyonly61%? Why cannot I getcompression rates over 90%as I commonly see with my hypertables?The reason behind this is that continuous aggregates store partial representation of the aggregates inbytea format, which is compressed using dictionary-based compression algorithms. So continuous aggregates cannot take advantage of other compression algorithms - at least, not yet. We’re experimenting with some concepts that may significantly increase the compression rates for continuous aggregates in the future. Stay tuned!Set up a data retention policy on your raw dataThrough the continuous aggregate we created earlier, we could effectively downsample our data. Our original dataset had one datapoint per minute; this is perfect for real-time monitoring, but a bit too heavy for the purpose of long-term analysis. In our continuous aggregate, we’re aggregating the data into 1 h buckets, which is a more manageable granularity if you’re planning to store this data long-term. So let’s save up some additional disk space by dropping the data in the underlying hypertable while keeping the continuous aggregate.To do so, we will define aretention policythat automatically deletes our raw data once it reaches a certain age. This age is completely up to you and your use case (the retention policy below will delete the data older than a month, for example). As a reminder, this policy refers only to the data in your original hypertable - you will still keep the data materialized in the continuous aggregate.SELECT add_retention_policy('stocks_intraday', INTERVAL '1 month');You asked, and we delivered: introducing timezones for continuous aggregatesAs we mentioned at the beginning of this post, TimescaleDB 2.6 introduces not only compression for continuous aggregates but also the possibility of using timezones in continuous aggregates,another highly requested feature by our community. Starting with TimescaleDB 2.6, you're able to create continuous aggregates with monthly buckets and/or timezones usingtime_bucket_ng.This is an experimental feature: please,test it and send us your feedback!The more feedback we get, the faster we will make this feature ready for production. 🔥time_bucket_ngis the “new generation” of ourtime_buckethyperfunction,  used for bucketing and analyzing data for arbitrary time intervals in SQL. You can think oftime_bucketas a more powerful version of thePostgreSQLdate_truncfunction, allowing for arbitrary time intervals rather than the standard day, minute, hour provided bydate_trunc.Buttime_bucketdoesn’t support time buckets by months, years, or timezones.time_bucket_ngexpandstime_bucketby including these features, giving users maximum flexibility in their queries. (We also first introducedtime_bucket_ngthanks to an issue from the community!)time_bucket_ngis still an experimental feature, being that it’s still being developed underour experimental schema. In Timescale, we like to bring constant value to our users by moving fast, but we also value stability - as we like to say, wemove fast without breaking things. Through our experimental schema, we’re allowed to release experimental features that, even if they’re still not ready for production, are ready to be widely tested. As we keep mentioning through this post, we love to get as much feedback from the community as possible: releasing features under experimental helps us build robust features, as by the time we “graduate” these features out of the experimental schema, we are sure that everything is stable.Starting with TimescaleDB 2.6, you can now use time buckets of months and years plus specify timezones in your continuous aggregates. For example, the continuous aggregate below tracks the temperature in Honolulu over monthly intervals.(Yes, we are dreaming of warmth 🏝 in this team!)CREATE TABLE conditions(
  day timestamptz NOT NULL,
  city text NOT NULL,
  temperature INT NOT NULL);

SELECT create_hypertable(
  'conditions', 'day',
  chunk_time_interval => INTERVAL '1 day'
);

INSERT INTO conditions (day, city, temperature) VALUES
  ('2021-06-14 00:00:00 HST', 'Honolulu', 26),
  ('2021-06-15 00:00:00 HST', 'Honolulu', 22),
  ('2021-06-16 00:00:00 HST', 'Honolulu', 24),
  ('2021-06-17 00:00:00 HST', 'Honolulu', 24),
  ('2021-06-18 00:00:00 HST', 'Honolulu', 27),
  ('2021-06-19 00:00:00 HST', 'Honolulu', 28),
  ('2021-06-20 00:00:00 HST', 'Honolulu', 30),
  ('2021-06-21 00:00:00 HST', 'Honolulu', 31),
  ('2021-06-22 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-23 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-24 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-25 00:00:00 HST', 'Honolulu', 32),
  ('2021-06-26 00:00:00 HST', 'Honolulu', 32),
  ('2021-06-27 00:00:00 HST', 'Honolulu', 31);

CREATE MATERIALIZED VIEW conditions_summary
WITH (timescaledb.continuous) AS
SELECT city,
   timescaledb_experimental.time_bucket_ng('1 month', day, 'Pacific/Honolulu') AS bucket,
   MIN(temperature),
   MAX(temperature)
FROM conditions
GROUP BY city, bucket;

-- to_char() is used because timestamptz is displayed in the sesison timezone by default
-- alternatively you can use SET TIME ZONE 'Pacific/Honolulu';
SELECT city, to_char(bucket at time zone 'HST', 'YYYY-MM-DD HH24:MI:SS') as month, min, max
FROM conditions_summary
ORDER by month, city;

   city   |        month        | min | max
----------+---------------------+-----+-----
 Honolulu | 2021-06-01 00:00:00 |  22 |  34
(1 row)We’re eager to know how this feature is working for you so we can improve it. Please, reach out to us throughGitHub, ourCommunity Slack, or theTimescale Community Forum.And thank you again for your invaluable support!Get startedTimescaleDB 2.6 is already available for Timescale Cloud and self-managed TimescaleDB:If you are a Timescale Cloud user, you will be automatically upgraded to TimescaleDB 2.6during your next maintenance window. No action is required from your side. You can also create a free Timescale Cloud account to get a free 30-day trial, with no credit card required.If you are using TimescaleDB in your own instances,check out our docs for instructions on how to upgrade.If you are using Managed Service for TimescaleDB, TimescaleDB 2.6 will be available for you in the upcoming weeks.Once you’re using TimescaleDB, connect with us! You can find us in ourCommunity Slackand theTimescale Community Forum. We’ll be more than happy to answer any question on continuous aggregates, compression, TimescaleDB, PostgreSQL, or anything in between.And if you want to help us build and improve features like compression, continuous aggregates, andtime_bucket,we are hiring broadly across many roles! Join our global, fully remote team. 🌎Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/increase-your-storage-savings-with-timescaledb-2-6-introducing-compression-for-continuous-aggregates/
2022-06-22T13:00:24.000Z,How We Fixed Long-Running PostgreSQL now( ) Queries (and Made Them Lightning Fast),"It was just another regular Wednesday in our home offices when we received a question in theforum. A TimescaleDB user with dozens of tables of IoT data reported a slow degradation in query performance and a creeping server CPU usage. After struggling with the issue, they turned to our community for help.via GIPHYThat same question came up in our forum,Community Slack, andsupportmore often than we’d like. We could relate to this particular pain point because we also struggled with it in a partitioned vanilla PostgreSQL. After a closer look at the user’s query, we found the usual suspect: the issue of high planning time in the presence of many chunks—in Timescale slang, chunks are data partitions within a table—and in a query using a rather common function:now().Usually, the problem with these queries is that the chunk exclusion happens late. Chunk exclusion is what happens when some data partitions are not even considered during the query to speed up the process. The logic is simple: the fewer data a query has to go through, the faster it is.However, the problem is thatnow(),similarly to other stable functions in PostgreSQL, is not considered during plan-time chunk exclusion, those precious moments in which your machine is trying to find the quickest way to execute your query while excluding some of your data partitions to further speed up the process. So, your chunks are only excluded later, at execution time, which results in higher plan time—and yes, you guessed it—slower performance.Until now, every time this issue popped up, we knew what to do. We had written a wrapper function, marked as immutable, that would call thenow()function and whose only purpose was to add the immutable marking so that PostgreSQL would consider it earlier during plan-time chunk exclusion, thus improving query performance.Well, not anymore.Today, we’re announcing the optimization of thenow()function with the release of TimescaleDB 2.7, which solves this problem by natively performing as our previous workaround.In this blog post, we’ll look at the basics of thenow()function, explain how it works in vanilla PostgreSQL and our previous TimescaleDB version, and wrap everything up with a description of our optimization and performance comparison that will blow you away (all we can say for now is “more than 400 times faster”).via GIPHYIf you are already a TimescaleDB user,check out our docs for instructions on how to upgrade. If you are using Timescale, upgrades are automatic, so all you need to do is sit back and enjoy this very fast ride! (New to Timescale?You can start a free 30-day trial, no credit card required.)now( ) in Vanilla PostgreSQLQueries withnow()expressions are common in time-series data to retrieve readings of the last five minutes, three hours, three days, or other time intervals. In sum,now()is a functionthat returns the current time or, more accurately, the start time of the current transaction. These queries usually only need data from the most recent partition in a hypertable, also called chunk.A query to retrieve readings from the last five minutes could look like this:SELECT * FROM hypertable WHERE time > now() - interval ‘5 minutes’;To understand our users' slowdown, it’s vital to know that constraints in PostgreSQL can be constified at different stages in the planning process. The problem withnow()is that it can only be constified during execution because the planning and execution times may differ.Sincenow()is a stable function, it’s not considered for plan-time constraint exclusion; therefore, all chunks will have to be part of the planning process. For hypertables with many chunks, this query's total execution time is often dominated by planning time, resulting in poor query performance.If we dig a little deeper with the EXPLAIN output, we can see that all chunks of the hypertable are part of the plan, painfully increasing it.Append  (cost=0.00..1118.94 rows=1097 width=20)
   ->  Seq Scan on _hyper_3_38356_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38357_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38358_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38359_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38360_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38361_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())We had to do something to improve this, and so we did.now( ) in TimescaleDBAs proud builders on top of PostgreSQL, we wanted to come up with a solution. So in previous versions of TimescaleDB, we did not use thenow()expression for plan-time constraint exclusion.In turn, we implemented constraint exclusion at execution time in a bid to improve query performance. If you want to learn more about how we did this,check out this blog post, which offers a detailed behind-the-scenes explanation of what happens when you execute a query in PostgreSQL.While the resulting plan does look much slimmer than the original, all the chunks were still considered during planning and removed only during execution. So, even though the resulting plan looks very different (look at those 1,096 excluded chunks), the effort is very similar to the vanilla PostgreSQL plan.Custom Scan (ChunkAppend) on metrics1k  (cost=0.00..1113.45 rows=1097 width=20)
   Chunks excluded during startup: 1096
   ->  Seq Scan on _hyper_3_39453_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())Close, but not good enough.now( ) We're TalkingWith our latest release, TimescaleDB 2.7, we approached things differently, adding an optimization that would allow the evaluation ofnow()expressions during plan-time chunk exclusion.Looking at the root of the problem, the reason whynow()would not be correct is due to prepared statements. If you executenow()but only use that value in a transaction half an hour later, the value does not reflect thecurrent time—now()—anymore.However,it will still hold true for certain expressions even as time goes by.For example,time >= now()will be true at this moment, in 5 minutes and 10 hours. So, when optimizing this, we looked for expressions that held as time passed and used those during plan-time exclusion.The initial implementation of this feature works for intervals of hours, minutes, and seconds (e.g.,now() - ‘1 hour’).As you can see from the EXPLAIN output, chunks are no longer excluded during execution. The exclusion happens earlier, during planning, speeding up the query. Success!Custom Scan (ChunkAppend) on metrics1k  (cost=0.00..1.02 rows=1 width=20)
   Chunks excluded during startup: 0
   ->  Seq Scan on _hyper_3_39453_chunk  (cost=0.00..1.02 rows=1 width=20)
         Filter: ((""time"" > '2022-05-24 12:41:31.266968+02'::timestamp with time zone) AND (""time"" > now()))In the next TimescaleDB version, 2.8, we are removing the initial limitations of thenow()optimization, making it also available in intervals of months and years. This means that you will be able to make the most of this improvement in a wider range of situations, as anytime > now() - Intervalexpression will be usable during plan-time chunk exclusion.Custom Scan (ChunkAppend) on metrics1k  (cost=0.00..1.02 rows=1 width=20)
   Chunks excluded during startup: 0
   ->  Seq Scan on _hyper_3_39453_chunk  (cost=0.00..1.02 rows=1 width=20)
         Filter: (""time"" > now())This code is alreadycommittedin ourGitHub repo, and will be available shortly.How Does It Work?But how did we make this current version happen? The optimization works by rewriting the constraint. For example:time > now() - INTERVAL ‘5 min’turns into((""time"" > (now() - '00:05:00'::interval)) AND (""time"" > '2022-06-10 09:58:04.224996+02'::timestamp with time zone))This means that the constified part of the constraint will be used during plan-time chunk exclusion. And, assuming that time only moves forward, the result will still be correct even in the presence of prepared statements, as the original constraint is ANDed with the constified value.Rewriting the constraint makes the constified value available to plan-time constraint exclusion, leading to massive reductions in planning time, especially in the presence of many chunks.So we know that this translates into faster queries. But how fast?Performance Comparison—now( ) That Is Fast!As shown in our table, the optimization’s performance improvement scales with the total number of chunks in the hypertables. The more data partitions you’re dealing with, the more you’ll notice the speed improvement—up to 401x faster in TimescaleDB 2.7for a total of 20,000 chunks when compared to the previous version.now()that is fast. 🔥The table lists the total execution time of the query (at the beginning of the post) on hypertables with a different number of chunksnow( ) Go Try ItThere are few things more satisfying for a developer than solving a problem for your users, especially a recurring one. Achieving such performance optimization is just the icing on the cake.If you want to experience the lightning-fast performance of PostgreSQLnow()queries for yourself, TimescaleDB 2.7 is available for Timescale and self-managed TimescaleDB.If you are a Timescale user, you will be automatically upgraded to TimescaleDB 2.7. No action is required from your side. You can also create a free Timescale account to geta free 30-day trial(no credit card required).If you are using TimescaleDB in your own instances,check out our docs for instructions on how to upgrade.Once you’re using TimescaleDB, connect with us! You can find us in ourCommunity Slackand theTimescale Community Forum. We’ll be more than happy to answer any question on query performance improvements, TimescaleDB, PostgreSQL, or other time-series issues.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-we-fixed-long-running-postgresql-now-queries/
2023-10-30T15:00:05.000Z,Migrating a Terabyte-Scale PostgreSQL Database to Timescale With (Almost) Zero Downtime,"(Almost) Zero Downtime Migrations (What, Why, and a Disclaimer)Zero downtime database migrations are the unicorn of database management: they’re legendary, but deep down, we know they don’t exist. Regardless of which vendor you’re migrating to, you will always have to reconnect to your target database after the migration, incurring a (preferably minimal) downtime period.With that disclaimer out of the way, database migrations also don’t have to be a stressful, gloomy process. They can be effective, secure, and straightforward.Today, we’re introducing live migrations, a battle-tested, (almost) zero downtime migration strategy for Timescale that will enable you to migrate your terabyte-scale PostgreSQL database effortlessly and safely—because the only thing better than a unicorn is a workhorse.And what’s best: in one fell swoop and using a few simple commands, you can migrate your data to Timescaleandenable one of our most popular features—hypertables—in your target database so that automatic partitioning is already running smoothly once your data comes in.Live migrations is the result of our extensive experience assisting our customers with the nerve-racking process of database migrations. It aims to simplify the transition process to Timescale, even if you're working with large PostgreSQL databases.In this article, we’ll go over some of the traditional PostgreSQL database migration challenges and explain why Postgres’ logical replication feature wasn’t enough before fully delving into our new, almost zero downtime migration strategy. This is thefirst announcement ofCloud Week, the first of three consecutive launch weeks here at Timescale—stay tuned!💡Ready to get started? You can find the step-by-step instructions for performing the live migrations described in this articlein our docs.The Challenges of Migrating Large Databases (A Time-Series Example)Let’s face it. Nobody wants to migrate their production database. But when your current PostgreSQL database no longer meets your application’s performance needs—even after some carefulperformance tuning—it’s time to move your data elsewhere.Data migrations can be challenging, especially when you have terabytes of data. The more data you have to move, the longer the migration will take. At the scale of terabytes, it could take days or weeks to transfer the data, which only compounds the issue because you can't take your application offline for that long (if at all).Plus, if you havetime-series datacoming at you fast and furiously, things only get more complicated. Most of our customers manage some kind of timestamped data (with all its specificities), so we’re particularly aware of these challenges.For example, picture an IoT application using PostgreSQL as its primary database for storing sensor data and the management platform’s operations. This includes device registration, sensor readings, alerts, and device configurations.This IoT management platform performs 10,000 inserts per second, 4,000 updates per second, and 2,000 deletes per second. Additionally, it serves approximately 20,000 queries per second due to different API calls happening on the front end based on the actions of end users. The existing database size is 2.7 TB, a substantial amount of data to migrate.Migrating 2.7 TB of data will take approximately 24-48 hours, depending on the network bandwidth, compute resources allocated in the source and target, and the machine where the migration occurs. As you can easily see, the main problem with this database migration is that downtime is not affordable, meaning the source database must remain uninterrupted until the migration is complete and the user is ready to make the switch.Additionally, the insert, update, and delete rate ranges from 10,000 to 2,000 per second—a significant amount of traffic and data operations occurring at the source. All these real-time operations must be replicated on the target end to ensure a smooth migration.With all these moving parts, a thoughtful data migration demands minimal downtime, avoiding missed sensor readings and an unavailable IoT management platform for end users. The ultimate goal is to ensure the service remains accessible and the migration runs seamlessly.Cue in live migrations: Traditionally, performing a migration while the data and operations are happening at the source would be concerning and sensitive. But, with our newest migration strategy (we have two others,pg_dump and pg_restore, anddual writes and backfill), the process becomes seamless because we do the heavy lifting for you.Why Doesn't Postgres’ Built-In Logical Replication Suffice?At this point, you’re probably wondering: “Wouldn’t it be simpler to use PostgreSQL’s native logical replication instead of creating a completely new migration strategy?” The short answer is “no.”PostgreSQL offers logical replication, a feature introduced in version 11, which aids in smooth migrations with minimal downtime. This feature operates on a PUBLISH/SUBSCRIBE model, with the source database publishing changes and subscribing to the target database.Timescale’s automatic partitioning on hypertables uses PostgreSQL’schild tables and inheritance. Each child table (we call them chunks) is assigned a time range and only contains data from that range.Normally, when you insert into the parent table, rows are written directly into the parent table, not the underlying child table. This is why TimescaleDB extends theplannerto control re-route inserts into a hypertable to the right chunk. Unfortunately, Postgres' native logical replication applies changes directly to the parent table, bypassing TimescaleDB's logic to route changes to the correct chunk.The Timescale Engineering team isworkinghard to address this with upstream Postgres, but it is still a work in progress and will take time to be generally available. In fact, this is one of the main reasons we developed live migrations: we didn’t want our customers to wait for a speedy, minimal downtime migration from PostgreSQL to Timescale, and our two existing migration strategies didn’t suit all of our customers’ use cases and requirements.How Are We Solving the PostgreSQL to Timescale Migration Puzzle?To create live migrations, we have developed a migration workflow for Timescale on top of pg_dump/pg_restore (for schema) and Postgreslogical decoding(for live data). Let’s explore some of the basic concepts involved:📖What is logical decoding?Logical decoding turns complex database changes into a simple format that's easy to understand without worrying about the database’s internal storage format. PostgreSQL achieves this by transforming the write-ahead log (WAL) into a more user-friendly format, like a series of data entries or SQL statements.What is write-ahead logging (WAL)?You probably know that Postgres' transactions are ACID. The D stands for ""durability"" and ensures that once a transaction has been committed, its changes persist, even in the face of failures. PostgreSQL manages durability through the WAL, which is an append-only, persistent data structure that logs all modifications made to the database. By recording these changes to a file, the database ensures that each modification remains durable and recoverable.💡 Tip:If you’re looking for more information on how to keep your data safe, be sure to check our article onPostgreSQL database backups.Logical decoding facilitates the subscription to real-time database changes. These changes are streamed to subscribers in accessible formats, such as JSON. Subscribers can then convert these changes to SQL statements and apply them to the target database.For our Timescale migration, we opted to leveragepgcopydb, a rising open-source tool in the Postgres community, rather than building our own logical decoding solution from the ground up.While pgcopydb can do the complete migration in one go, we decided to use it only for historical data migration and live data replication. We wanted finer control over the schema migration process to enable features likehypertables(a Timescale feature that automatically partitions your data).After the migration, you can still enable hypertables. However, converting non-emptyplain tables to hypertablescan lock the table for a significant amount of time, preventing any modifications to the table until the conversion is complete. This means that no new data can be written into the table while the conversion is in progress. The duration of the lock increases with the size of the table.The Live Migrations ProcessAs mentioned, live migrations leverages logical decoding using pgcopydb to ensure changes made to the source database are applied to the target database.For example, when you use pg_dump to take a snapshot of the data in the source database, all changes made in the source after the pg_dump command are lost. With live migrations, these changes are stored and later replayed into the target database with the help of pgcopydb.How live migrations ensures the safe migration of terabytes of data from PostgreSQL to TimescaleTransactional consistencyensures that all operations within a transaction are either completed successfully or not at all. If a transaction is interrupted, such as due to a system crash or power outage, any changes made during that transaction are not saved, and the system returns to its original pre-transaction state.Thus, transactional consistency refers to the guarantee that the state of the target database will reflect the state of the source database from a specific point in time. When migrating data, it is important that the source and target databases remain consistent with each other. This means the following:Completeness: If a transaction was committed on the source database and affected multiple rows or tables, those changes should be entirely replicated in the target database.Order preservation: The order of transactions in the source should be preserved in the target. If transaction A is completed in the source before transaction B, the same order should be reflected in the target.Transactional consistency is crucial for data integrity, preventing data anomalies and ensuring the migrated database remains a true and reliable reflection of the source database.To maintain transactional consistency, the live migrations process involves moving the schema first, followed by enabling hypertables. Hypertables are Timescale's method of partitioning tables for optimal performance. If you want to learn more abouthypertables and Postgres partitioning,check out this article.Once the database schema has been migrated and enabled with hypertable settings for the desired plain tables, logical decoding is initiated to subscribe to all real-time operations happening at the source. These operations are stored in an intermediate storage running pgcopydb until the data backfilling is complete.After initiating logical decoding, the data backfilling process begins. Once the data backfilling is successful, the real-time transactions stored in the intermediate storage are applied to the target database, keeping the source and target in sync in real time without the need for intermediate storage.The time it takes for the source and target to be fully synchronized can vary, depending on the ingest load at the source. When the target catches up with the source, you can observe in the pgcopydb logs that the target is a few seconds to minutes behind the source. At this point, you can perform data integrity and verification checks.Once you are confident with the move, switch your applications to write to the target database. This will achieve near-zero downtime but may cause transactional inconsistency between the lag the migration is trying to catch up with and the latest changes made by your application to the target.If the application's changes on the target do not depend on the latest few minutes that the migration is trying to catch up with, you can proceed with the migration by making the target the primary and discontinuing writes to the source.However, if you prioritize transactional consistency, you will need to stop writes to the source and wait until the lag between the source and target reaches zero. Once the lag is eliminated, you can resume writes to the target. As writes to the source have been halted during the switch, pgcopydb will catch up with the few seconds to minutes of delay and indicate that the migration is in sync. At this stage, you can safely conclude the migration process since there will be no further changes to replicate from the source to the target.Performance and reliabilityThe performance and reliability of pgcopydb are improving day by day. We have contributed fixes to improve itsINSERT throughputandreliabilityin live data replication mode, as well as fixingmemory leaks. Other performance improvements, like usingCOPYandpipeline mode, are currently underway.For the historical data copying, pgcopydb uses the standardPostgres COPY protocolto stream the data from source to target. Pgcopydb not only supports parallel copying of multiple table data at once but also supportssame table concurrencyby splitting a table into multiple parts to accelerate the data movement and reduce the duration of the migration window.Migration is inherently sequential: any changes to the source database must be buffered until the historical data transfer is completed. While typical logical replication tools likePostgres native replicationandpglogicalretain the WAL segments on the source database until the changes are consumed, this can result in excessive storage consumption and potential database failures.The creators of pgcopydb took aunique approachand chose to buffer the live changes into the volume attached to the computer where it runs. This grants users increased flexibility, allowing them to opt for larger storage capacities or even resort tocloud-based storage solutions like AWS S3.Migrating a Large Postgres Workload to TimescaleNow that we’ve explained our live migration solution in detail, let’s see how you can actually move a large PostgreSQL workload to Timescale without batting an eye.Let’s use our earlier IoT example—remember the 10,000 inserts per second? Suppose we're storing them in areadingstable ingesting data from various sensors, which, over time, is bound to grow because the developer wants to keep that data for a while. Without appropriate partitioning, this table’s insertions and queries could slow down.So, to tackle this issue, let’s migrate the data to Timescale. By doing so, we gain not only the advantages of automatic partitioning but also the perks of a managed service.Before diving in, ensure the following:For optimal performance, run pgcopydb on a machine geographically near both your source and target to minimize network latency. For example, if your databases are hosted in the us-east-1 region, consider provisioning an EC2 instance in the same region. Ensure the instance is equipped with a minimum of 4 CPUs and 4 GB RAM for smooth operation.The amount of disk required for pgcopydb to operate is a function of the time required to transfer historical data and the rate of DML operations executed on the source database. To illustrate, with an insert rate of 10 K/s, pgcopydb writes about 6.5 MB/s to storage. This equates to around 23 GB every hour or 550 GB daily. If the transfer of historical data spans two days, the disk should have a minimum capacity of 1,100 GB.Install pg_dump, psql, and pgcopydb on the computer you're using for the migration. If you are using a Debian-based distro, run the following to install the tools:sudo apt-get install postgresql-client pgcopydbThe source database must have “wal_level” set to “logical.” You can check by running the following command on the psql prompt connected to the source database.psql ""postgres://<user:password>@<source host>:<source port>"" -c ""SHOW wal_level"";

 wal_level
-----------
 logicalThe Postgres user must have a REPLICATION attribute set on their role to create a replication slot in the source database. You can check this by running the following command on the psql prompt connected to the source database.psql ""postgres://<user:password>@<source host>:<source port>"" -c ""\du <user>"";

 List of roles
 Role name |                   Attributes                    |                         Member of
-----------+-------------------------------------------------+-----------------------------------------------------------
 tsdbadmin | Create role, Create DB, Replication, Bypass RLS | {pg_read_all_stats,pg_stat_scan_tables,pg_signal_backend}To simplify the migration process, it's a good practice to set up your source, target database connection strings, and pgcopydb working directory as environment variables. This way, you don't have to re-enter them every time you use a tool, reducing the risk of errors.On most systems, you can set an environment variable using the command line:export SOURCE=postgres://<user:password>@<source host>:<source port>
export TARGET=postgres://<user:password>@<target host>:<target port>

export PGCOPYDB_DIR=/path/to/large/volume/mountPGCOPYDB_DIRThis is the directory where pgcopydb retains its operational data, including the buffered live replication data during the change data capture phase.1. Begin logical decodingKickstart the migration by running pgcopydb in follow mode. It does the following:It creates areplication slotin the source database, which represents a stream of changes that can be replayed to a client in the order they were made on the origin server. The changes emitted by the slot are buffered into the disk.It creates and exports asnapshotin the source database to migrate the data that exists in the database (historical data) prior to the creation of the replication slot. The exported snapshot will be available in the$PGCOPYDB_DIR/snapshotfile.After creating the replication slot and snapshot, it will automatically start buffering changes happening in the source to the intermediate files under the$PGCOPYDB_DIR/cdcdirectory. Thepgcopydb followcommand only takes care of the live data replication part; still, schema & historical data have to be migrated separately, which we discuss in the next section.pgcopydb follow \
  --dir ""$PGCOPYDB_DIR""
  --source ""$SOURCE"" \
  --target ""$TARGET"" \
  --fail-fast \
  --plugin wal2json > pgcopydb_follow.log 2>&1 &The above command is going to be active during most of the migration process. You can run it on a separate terminal instance or start it in the background. To start it in the background, append> pgcopydb_follow.log 2>&1 &to redirect all the messages to thepgcopydb_follow.logfile, this is optional but recommended. Thepgcopydb followcommand outputs many messages, if they are not redirected, using the terminal becomes cumbersome due to the constant pop-up of messages.Important: To replicate DELETE and UPDATE operations, the source table must haveREPLICA IDENTITYset. This assists logical decoding in identifying the modified rows. Although it defaults to using the table's PRIMARY KEY as REPLICA IDENTITY, if no PRIMARY KEY is available, you'll need to manually set REPLICA IDENTITY to enable replication of UPDATE and DELETE operations.2. Migrate schemaUse the pg_dump command to dump only the schema without any data.pg_dump -d ""$SOURCE"" \
  --format=plain \
  --quote-all-identifiers \
  --no-tablespaces \
  --no-owner \
  --no-privileges \
  --schema-only \
  --file=dump.sql \
  --snapshot=$(cat ${PGCOPYDB_DIR}/snapshot)Apply the schema dump into the TARGET Database:psql ""$TARGET"" -v ON_ERROR_STOP=1 --echo-errors \
	-f dump.sqlThis is the ideal point to convert regular tables into hypertables. In simple terms, you might want to convert the tables that contain time-series data to leverage Timescale’s automatic time-based partitioning. For instance, you can set up thereadingstable as a hypertable by running the following command.psql ""$TARGET"" -c ""SELECT create_hypertable('readings', 'time', chunk_time_interval=>'1 week'::interval);""You may want to repeat this for all the candidates who can benefit from Timescale’s automatic partitioning.Note:The conversion of the normal table into a hypertable has the downside of locking the table until the conversion completes. The larger the table, the longer it is going to remain locked, and it doesn’t allow any modification to the table during that time.3. Migrate historical dataThe following command migrates the historical data from source to target using the efficientPostgreSQL COPY command. It not only can copy multiple tables in parallel, but it can also split a single large table into multiple smaller parts and copy those in parallel too. (See--split-tables-larger-than).pgcopydb copy table-data \
 --dir ""$PGCOPYDB_DIR""
  --source ""$SOURCE"" \
  --target ""$TARGET"" \
  --snapshot $(cat ${PGCOPYDB_DIR}/snapshot) \
  --split-tables-larger-than 10G  \
  --table-jobs 8By using the snapshot created in step 1, it guarantees that only the data that existed on the database prior to the creation of the replication slot is copied.Depending on the historical data size, the command takes anywhere from a few hours to several days. While it's copying,pgcopydb follow,which we started in step 1, will keep track of any new changes made in the source database and save them to disk.4. Apply the replication changesAfter the completion of historical data migration, instruct pgcopydb to start applying the buffered changes stored in the file, followed by live data replication.pgcopydb stream sentinel set apply --source ""$SOURCE""The replication progress can be monitored by querying thepg_stat_replicationview like:psql ""$SOURCE"" \
  -f - <<'EOF'
SELECT write_lag, flush_lag, replay_lag
FROM pg_stat_replication
WHERE application_name='pgcopydb' AND state='streaming'
EOFIt shows three different values:write_lag: the time elapsed between flushing recent WAL at SOURCE and receiving notification that the pgcopydb has written it into the disk.flush_lag: the time elapsed between flushing recent WAL at SOURCE and receiving notification that the pgcopydb has written and flushed it into the disk.replay_lag: the time elapsed between flushing recent WAL at SOURCE and receiving notification that the pgcopydb has written, flushed it into the disk and applied at the TARGET.We are more interested inreplay_lagbecause, as described, it shows the replication lag in thetimeduration between the source and the target. In an ideal scenario, thereplay_lagmust come down gradually over time and should oscillate between a few seconds to minutes.If you notice the replay_lag is getting longer, this might mean:The target database isn't working.The target database is struggling to keep up with the source due to limited resources.5. Prepare to switchoverIf all goes as well and once thereplay_lagcomes down to a few seconds, It is time to update target table statistics to get better query performance after the switchover.psql $TARGET -c ""ANALYZE;""Upon completion ofANALYZE,it is recommended to inspect the target database by executing a few queries used in your application to verify data integrity, even if the target is slightly behind (a few seconds to minutes) the source. Performing these checks prior to the switchover would avoid last-minute surprises after promoting the target database as the primary.Once the preliminary inspections on the target are positive, it is time tostop the write traffic to the source databaseso that the pending changes can be applied to make both the source and target identical.With no new data being ingested into the source, the replication slot should not emit any more changes, andreplay_lagshould decrease considerably faster.Let´s instruct pgcopydb to stop when it's done applying the changes that have been generated up to this point:pgcopydb stream sentinel set endpos --current --source ""$SOURCE""Postgres logical decoding doesn’t replicatesequences, The following command copies the latest values of all sequences from source to target.pgcopydb copy sequences --dir ""$PGCOPYDB_DIR"" \
  --source ""$SOURCE"" \
  --target ""$TARGET"" \
  --resume \
  --not-consistentWait for thepgcopydb followprocess from step 1 to finish its execution. The process runs until the end position is reached. If you startedpgcopydb followin the background, you can bring it to the foreground with thefgcommand.A successful execution of thepgcopydb followcommand should have logs stating that the end position has been reached and that the process is done.07:58:28.859 119 INFO   Transform reached end position 0/2ECDB58 at 0/2ECDB58
07:58:29.108 120 INFO   Replay reached end position 0/2ECDB58 at 0/2ECDB58
07:58:29.168 120 INFO   Replayed up to replay_lsn 0/1AB61F8, stopping
07:58:29.473 8 INFO   Subprocesses for prefetch, transform, and catchup have now all exited
07:58:29.534 8 INFO   Current sentinel replay_lsn is 0/2ECDB58, endpos is 0/2ECDB58
07:58:29.534 8 INFO   Current endpos 0/2ECDB58 has been reached at 0/2ECDB58
07:58:29.534 8 INFO   Follow mode is now done, reached replay_lsn 0/2ECDB58 with endpos 0/2ECDB58If the command output was redirected to a file, the messages won't be shown in the terminal even if you bring the process to the foreground. In this case, inspect the log file.Now that all data has been moved, the contents of both databases should be the same. How exactly this should best be validated is dependent on the application. One option is to compare the number of rows in the source and target tables, although this reads all data in the table, which may have an impact on the production workload.The final step is to switch to the target database as your primary by changing the application’s connection string to point to the target instance.As the last step, clean up thepgcopydbartifacts:pgcopydb stream cleanup --source ""$SOURCE"" --target ""$TARGET""The future of live migrationsAs you’ve probably noticed by now, we’ve put a lot of effort into our live migrations solution, but that doesn’t mean the work is done yet.Currently, our migration workflow using pgcopydb only supports migrations from PostgreSQL to Timescale. However, we are in the process of enabling Timescale to Timescale live migrations as well. We are also actively working with the pgcopydb community to further improve theperformanceandreliabilityaspects of the tool.In addition, we are enhancing the pre-migration and post-migration checks to streamline the end-to-end migration process. This will simplify the start of the migration process and make it easier to verify and switch the primary database to Timescale.All of these enhancements are part of our overarching goal for this solution: we know how precious your data is and want to ensure a seamless and worry-free transition.If you are looking to migrate your data from self-hosted TimescaleDB to Timescale, a fully managed cloud platform that makes PostgreSQL faster and fiercer,reach out! We’ll be happy to assist you. For feedback on live migrations, hit us up onSlack (#migrations channel) and be a part of our community. We'd love to hear from you!Next stepsLive migrations is just the beginning of easing the migration process to Timescale. We are actively iterating on our solutions to ensure they are simple and effective, enabling data migrations (including heavy workloads) with just a few clicks and providing best-in-class performance and reliability for future migrations.If you would like to move your data using live migrations (or one of our other migration strategies), please follow the Timescale documentation:pg_dump and pg_restore for Postgres to TimescaleDual writes and backfill for Postgres to TimescaleLive migrations for Postgres to TimescaleTo try Timescale,sign up for a free trial (no credit card required). Timescale includes our core database, TimescaleDB, but enhances it with features designed for higher performance, faster queries, and cost savings.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/migrating-a-terabyte-scale-postgresql-database-to-timescale-with-zero-downtime/
2022-09-29T12:57:19.000Z,What’s New in TimescaleDB 2.8?,"TimescaleDB 2.8 is now available onTimescaleand fordownload. This major release includes the following new features:Thetime_buckethyperfunction now supports bucketing by month, year, and time zone, enabling easier time-based queries and reporting.You can now manage refresh, compression, and retention policies for continuous aggregates all in one easy step.Improved performance of bulkSELECTstatements that fetch high volumes of data from distributed hypertables using theCOPYprotocol.Support forON CONFLICT ON CONSTRAINT UPSERTstatements on hypertables, which enables better compatibility with some GraphQL/ PostgreSQL middlewares.Let’s explore these improvements in more detail.Thetime_buckethyperfunction now supports bucketing by month, year, and time zone“While the newtime_bucketfunctionality is rather simple, its impact is immense and simplifies analytical requests with the need of time zone information.”Chris Engelbert, “Nightmares of Time Zone Downsampling: Why I’m Excited About the Newtime_bucketCapabilities in TimescaleDB”Support for time zones and bucketing your data by monthly or yearly intervals using thetime_buckethyperfunctionwere some of the most requested features, and they’re finally available in TimescaleDB 2.8.Developed with thecommunity's helpover the last couple of months, the new implementation initially lived in TimescaleDB’sexperimental namespaceastime_bucket_ng. The functionality has since been improved through testing and feedback and is now generally available as part of thetime_buckethyperfunction. These capabilities simplify analytical requests with the need for time zone information. By providing atime zoneparameter in thetime_buckethyperfunction, developers can now adjust theoriginaccording to the given time zone. That means that those daily, monthly, or yearly boundaries are also modified automatically. When migrating existing code, all your current queries will work just as they did before the change.Using monthly or yearly buckets and specifying a time zone is simple, as illustrated in the query below:SELECT
   time_bucket('1 month', created, 'Europe/Berlin') AS bucket,
   avg(value)
FROM metrics
GROUP BY 1📚Read more aboutimprovements to thetime_buckethyperfunction.Create and manage multiple policies for continuous aggregates in one stepContinuous aggregateshelp developers query large amounts of time-series datamore quickly, and are one of the hallmark features of TimescaleDB. Prior to TimescaleDB 2.8, you could only add one policy at a time, which could be tedious and complicated. Now with TimescaleDB 2.8’s one-step policy for continuous aggregates, you can add, remove, or modify multiple policies of continuous aggregates with a single command, including refresh, compression, and retention policies. Note that this feature is experimental, so we welcome your feedback on how to improve it for production use.Watch thedemo videoor read ourdocumentationto learn more about one-step policy management for continuous aggregates.Performance improvements for distributed hypertables using theCOPYprotocol for high data volume queriesIn TimescaleDB 2.8, we improved the performance of queries that select a high volume of data from distributed hypertables, such asSELECT *.... For this, we started to use the so-calledCOPYprotocol—the subset of the PostgreSQL native protocol used to efficiently transfer data in bulk. We also made our code more efficient and reduced overhead by streamlining how we work with the internal data structures used to hold and transfer the row data.Hypertables now supportON CONFLICT ON CONSTRAINT UPSERTstatementsWith the TimescaleDB 2.8 release,hypertablesnow support theON CONFLICT ON CONSTRAINTclause, which fixes a long-standing compatibility issue with GraphQL/PostgreSQL middlewares like Hasura, Prisma, and Postgraphile.Try TimescaleDB 2.8If you are using Timescale, upgrades are automatic, and you’ll be upgraded to TimescaleDB 2.8 in your next maintenance window.New to Timescale?Start a free 30-day trial, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/whats-new-in-timescaledb-2-8/
2023-11-02T11:30:00.000Z,"Connection Pooling on Timescale, or Why PgBouncer Rocks","Today, Timescale announces the general availability of connection pooling on our platform.Connection poolers help scale connections and get better performance out of their database, especially for customers with many short-lived connections.Unlike many competitors, the connection pooler, hostedPgBouncer, has dedicated infrastructure rather than being colocated with your database. Dedicated infrastructure means the pooler does not use any of the compute resources reserved for your database.💡You can add a connection pooler via the ""Connection info"" panel of your service overview in the Timescale console. If you are not a Timescale user,we offer a 30-day free trial, no credit card required.Scaling database connections is a long-standing problem in PostgreSQL. To address this,we are now offering connection pooling, which is effectively hosted PgBouncer, for our customers. This allows you to add a connection pooler to your service. Doing so gives you an additional port in your service connection string where you can connect to your service through the pooler.As part of the pooler, you get two pools: one session pool and one transaction pool. This allows you to potentially increase performance and throughput to your service via the transaction pool while reserving session connections for trusted clients. Since we are offering PgBouncer, you get all of the amazing benefits PgBouncer has to offer, including all of the monitoring via theSHOWcommands.If you have not considered a connection pool, you may want to. Adding a connection pool can improve your performance, especially when using transaction mode, even if you don’t need the additional connections.This is especially true for applications that are serverless in nature or rely on AWS Lambda, like many IoT applications, or that open and close connections often, like web applications. Opening and closing connections in PostgreSQL, as we’ll explain below, creates additional strain and overhead in your database, which can be alleviated by using a connection pooler.Besides the benefits of using a connection pooler, which you may already be familiar with, we will also talk about some of the engineering design considerations and decisions that were made as we added pooling as a part of our infrastructure.Connection Pooling and PostgreSQL PrimerHere, I’ll give a high-level overview of connection poolers and why they can be useful in PostgreSQL. If you are already familiar with connection pooling and are more interested in how we implemented it, you can probably safely skip this section. (I likethis articlefor a more in-depth explanation.)Connection pooling, at a basic level, is essentially a proxy that sits between your database and your application. This proxy (the pooler) maintains persistent connections with the database as part of a connection “pool.” When the application requests a connection, the pooler lets it use one of the connections from the pool. When the application is done with the connection, the pooler adds the connection to the database back to the pool, freeing it up for when the application needs another connection.This is fairly straightforward but doesn’t seem to provide much utility at face value and adds a layer between your application and the database. So, why do people want this? To answer that, we need to dive a bit into how PostgreSQL handles connections.Every time Postgres receives a request for a new connection, the backend process is forked, creating a new backend process for that connection and allocating approximately 5 MB of memory for each connection. Some internal actions often performed by the Postgres server are linear with respect to the number of client backends. The higher the number of backends (connections), the longer this process can take.Additionally, every time a new connection is opened, it takes time for the client and server to establish connections. That time overhead is wasted time within the database, especially for applications that are opening connections for very short sessions. Basically, the more connections you have, the more work you give Postgres to do for backend operations. (This can be particularly bad if there is a bug in the application that forgets to close connections, resulting in hundreds of concurrent backends!)Postgres connections: Less is moreIn short, connections in Postgres are relatively expensive, and Postgres prefers fewer long-lived connections. Postgres also does not maintain a queue of connections if there are already the maximum number of available connections created and instead throws an error.These approaches can directly conflict with modern application development, where one often opens the connection as late as possible and closes it as soon as possible and may anticipate a queue if there are no available connections. This is where connection pooling comes in!With connection pooling, Postgres gets what it prefers for optimal performance—fewer, long-lived connections—while the application can attempt to open many connections, and as often as it needs, without restraint. The connection pooler manages this conflict seamlessly and can maintain a queue of connections in the case where all of the connections in the pool are taken.This allows you to use Postgres in serverless, web, and IoT applications seamlessly, as these types of applications often prefer short-lived connections, which connection pooling can provide, and increases the chance of query success by being able to wait for a connection rather than have to manage an out of connections error.Connection pooling also helps scale applications that need to run multiple transactions per second. As the Postgres work model is a connection (session) per client, those applications have to either require Postgres to be configured with way more connections that it can handle efficiently or establish and close connections multiple times per second.The transaction mode provides a better alternative to both of those by sharing a single connection between multiple clients, as long as each client issues transactions (which may consist of a single statement) relatively independent of each other. With that, one can achieve many thousands of transactions per second with relatively few database connections.Transaction mode also benefits when applications are improperly configured and do not preemptively close the connection when they are done with it. In session mode, or without a pooler, this takes an entire connection slot. In transaction mode, this isn’t an issue, as the pooler releases the connection once the transaction has been completed.“My application handles connections perfectly and doesn’t have issues, so I don’t need transaction mode.” Great! Many connection poolers let you have multiple pools. You can create a session pool for trusted clients and a transaction pool for everyone else. Win/win!In summary, connection pooling in a Postgres context allows you to scale connections more easily and ensure your database will not pay with performance for a growing number of connections.Evaluation of PgBouncer and AlternativesAs we started to look into connection pooling, we considered several options on the market. We knew we likely did not want to build our own, as there are many incredible open-source options. We would rather leverage an existing, tried and tested option and contribute back where we can, rather than start from scratch. This likely won’t surprise many of you, given Timescale’s deep love for PostgreSQL!We considered many poolers. This includes Yandex’sOdysseyandpgpool-ii. Ultimately, though, we knew we probably wanted to go withPgBouncerorpgCat. PgBouncer has become the industry standard for connection pooling and is incredibly well respected in the industry. It’s known for its feature completeness and being incredibly reliable. It also uses few resources while being very fast, even while handling lots (10K+) of connections on a single process. For the features we were looking for in a pooler, PgBouncer was perfect.pgCat is an incredibly interesting alternative. It closely mirrors PgBouncer functionality, while being written in Rust. Going with pgCat could give us much more by offering additional features like load balancing, mirroring, and multi-threading. While these things are incredibly tempting to play around with, they’re a bit beyond the scope of what we are hoping to offer for customers.One thing pgCat offered that PgBouncer did not until recently was prepared statement support in transaction mode. However, with PgBouncer 1.21, that’s no longer the case!Given the limited gap in functionality between PgBouncer and other poolers, we ultimately decided to use the tried and true choice—PgBouncer. That said, how we’ve architected poolers could allow us to switch to pgCat (or another pooler) seamlessly in the future. We chose to run PgBouncer in its own pod. This allows us to perform maintenance on PgBouncer without impacting the database at all.If we colocated PgBouncer and the service, any time we wanted to make an update to just PgBouncer, we would need to restart the whole pod, including the customer database! Since PgBouncer is very lightweight, any changes to the pooler take place almost instantaneously.How to See Timescale Connection Pooler StatisticsWith PgBouncer, all connection pooling customers get access to all of the really useful read-onlycommand line commandsby default. This can be a great way to examine the statistics of your pooler. This can be particularly useful after adding it to ensure you’ve configured your connections properly and are connecting to the pools you’d like to.For example, to use theSHOWcommands:First, connect to your database via the pooler.Then, switch to the PgBouncer database, e.g.,\d pgbouncer.From there, you can simply runSHOWwith whatever you’re interested in, e.g.,SHOW STATS.Simple as that!A really common use case of this is to connect and run theSHOW POOLScommand. This tells you the number of database connections used, the number of clients that are active (which is typically way more than the number of database connections, especially in the transaction pool), and the number of clients waiting for a connection. This lets you see firsthand how much PgBouncer is helping your database scale client connections.How to add a connection pool to your serviceYou can add a connection pooler to your service in one of two ways.During service creation, you can toggle “enable connection pooler” in the service creation screen.Enabling PgBouncer in the Timescale console, in the service creation screenFor an already running service, you can add (or remove) a connection pooler at any time from the “Connection info” tab of the service overview by clicking “Connection pooler” and then “+ Add connection pooler.”Adding or removing a connection pooler in the Timescale console using the ""Connection info"" tabNext StepsWe are excited to offer connection pooling on Timescale! If you want to try connection pooling yourself, you can add a pooler to your service by following the steps above. If you’re not a customer,Timescale offers a 30-day free trial, no credit card required!For any questions or feedback, don’t hesitate to reach out. You can submit a support ticket ([email protected]) or contact me via ourcommunity Slack(@Grant Godeke).And don't forget to check our other launches during Cloud Week,the first of three consecutive launch weeks here at Timescale. So far, we've released:Live migrations: a new way to migrate your multi-terabyte PostgreSQL database to Timescale with (almost) zero downtime.Timescale Enterprise Tier: a solution designed for your demanding data and business needs.Insights: Experience in-depth database monitoring with Insights.We're just getting started—stay tuned for more!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/connection-pooling-on-timescale-or-why-pgbouncer-rocks/
2022-05-16T13:09:53.000Z,Observability Powered by SQL: Understand Your Systems Like Never Before With OpenTelemetry Traces and PostgreSQL,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Troubleshooting problems in distributed systems built on microservices and cloud-native technologies is a challenge, to say the least. With so many different components, all their interactions, and frequent changes introduced with autoscaling and multiple deployments, we can’t possibly predict everything that could go wrong.So far, we know that the first step to solving this problem is to start collecting detailed telemetry data from all our cloud-native systems in the form of traces, metrics, and logs. Unlike monoliths, troubleshooting distributed systems is a “trace first, metrics and logs second” process.By themselves, metrics and logs don’t provide the level of understanding required to identify anomalous behaviors in those systems. Traces, however, capture a connected view of how the entire system works to process requests, allowing software developers to figure out what isactuallyhappening when a problem occurs. Metrics and logs expand the information that traces provide—especially if directly correlated to traces.OpenTelemetry, a Cloud Native Computing Foundation (CNCF) standard for instrumentation, is driving a revolution in this space by unifying the collection of traces, metrics, and logs and making them universally available. Since OpenTelemetry focuses on helping solve the challenges of operating distributed systems, it is no surprise that the observability framework first targeted tracing support, which is currently more mature than the support for the other two signals.Having access to all that telemetry data—especially traces—solves only half of the problem.The other half requires tools to analyze, correlate and make sense of that data. That is what observability is about. It’s essentially an analytics problem: you want to be able to interrogate telemetry data with arbitrary questions and get results back in real time.Given that observability is an analytics problem, it is surprising that the current state of the art in observability tools has turned its back on the most common standard for data analysis broadly used across organizations: SQL.Good old SQL could bring some key advantages: it’s surprisingly powerful, with the ability to perform complex data analysis and support joins; it’s widely known, which reduces the barrier to adoption since almost every developer has used relational databases at some point in their career; it is well-structured and can support metrics, traces, logs, and other types of data (like business data) to remove silos and support correlation; and finally, visualization tools widely support it.However, implementing full SQL support from scratch on top of an observability store is a major undertaking that would likely take years of development.But what if we didn’t have to? What if we could leverage a well-known and reliable open-source SQL database to solve this problem?That’s exactly what we’ve done :)PostgreSQL isthe second most used and most loved databaseamong developers, backed by more than 25 years of development and a healthy and thriving community with hundreds of integrations available.You're probably thinking that observability data is time-series data that relational databases struggle with once you reach a particular scale. Luckily, PostgreSQL is highly flexible and allows you to extend and improve its capabilities for specific use cases.TimescaleDB builds on that flexibility to add time-series superpowers to the databaseand scale to millions of data points per second and petabytes of data.With PostgreSQL and TimescaleDB, we have a robust and scalable database for time-series data. Now, how do I add my OpenTelemetry data to start using SQL for observability?Promscale bridges that gap.Today, we are announcing the general availability of OpenTelemetry tracing support inPromscale, the observability backend for metrics and traces built on the rock-solid foundations of PostgreSQL and TimescaleDB. We’re now closer to becoming the unified data store for observability powered by SQL, bringing the PostgreSQL and observability worlds together.Traces are the first step toward truly understanding cloud-native architectures. They capture a connected view of how requests travel through their applications, helping developers understand (in real time) how the different components in their system behave and interact with one another.With today’s announcement of tracing support, Promscale equips software engineers worldwide withobservability powered by SQLfor distributed systems, allowing them to unlock unprecedented insights about their systems.OpenTelemetry Tracing: Observability Powered by SQLWith tracing support, engineers can interrogate their observability data with arbitrary questions in SQL and get results back in real time to identify production problems faster and reduce mean time to repair (MTTR).OpenTelemetry tracing support in Promscale includes all these features:A native ingest endpoint for OpenTelemetry traces that understands the OpenTelemetry Protocol (OTLP) to ingest OpenTelemetry data easilyTrace data compression and configurable retention to manage disk usageFull SQL superpowers to interrogate your trace data with arbitrary questionsA fully customizable, out-of-the-box, and modern application performance management (APM) experience in Grafana using SQL queries on OpenTelemetry traces, to get new insights immediately after deploymentOut-of-the-box integrations to visualize distributed traces stored in Promscale using Jaeger and Grafana, so you don’t have to learn new tools or change existing workflowsSupport for ingesting traces in Jaeger and Zipkin formats via the OpenTelemetry Collector, so you can also benefit from all these new capabilities even if you’re not yet using OpenTelemetryCorrelation of Prometheus metrics and OpenTelemetry traces stored in Promscale throughexemplarsas well as SQL queriesIntegration intobs, the observability stack for Kubernetes, so you can deploy all those new capabilities with a single command in your Kubernetes clusterKeep reading this blog post for more details on these capabilities and an introduction to powerful, pre-built Grafana dashboards that you can integrate into your Grafana instance. These dashboards will give you instant information on your distributed systems’ dependencies, helping you quickly identify (and correct) performance bottlenecks.You candeploy Promscale in your environment today—it’s 100 % free—or experiment with it byrunning our lightweight OpenTelemetry demo. To learn how to deploy a microservices application and a complete OpenTelemetry observability stack running on Promscale in only a few minutes,check our OpenTelemetry demo blog post for a step-by-step guide.Sending and Storing Trace Data in PromscaleA key advantage of OpenTelemetry is that it is vendor agnostic. This allows us to instrument our code with OpenTelemetry libraries and send the telemetry generated to one or multiple observability backends via exporters. OpenTelemetry also defines a line protocol (OTLP). All the OpenTelemetry language SDKs include an exporter to send telemetry data to OTLP-compliant backends. A number of other exporters have been built by the community and vendors to send the data to non-OTLP-compliant backends.Promscale has an endpoint that listens for OTLP over Google Remote Procedure Calls (by default on port 9202). So, it can ingest traces using the standard OpenTelemetry tools without needing a proprietary exporter.OpenTelemetry also includes the OpenTelemetry Collector, which is a component that allows it to receive telemetry in multiple formats (not just OTLP), process it, and export it via OTLP with the OpenTelemetry exporter or in other formats through proprietary exporters.Architecture diagram illustrating the core components of the OpenTelemetry collector, the inputs it accepts, and possible outputsTo send trace data from your application instrumented with OpenTelemetry to Promscale, you have two options:You can configure the OTLP exporter of the OpenTelemetry SDK you used to instrument your application to send traces directly to Promscale.You can configure the OTLP exporter of the OpenTelemetry SDK to send data to the OpenTelemetry Collector and then from the OpenTelemetry Collector to Promscale.We advise you to use the OpenTelemetry Collector for better performance because it can send data to Promscale in larger batches, which speeds up ingestion. Also, if you want to send data to another backend, you can change the configuration in one place.See our documentation for more details.You can also easily sendJaegerandZipkintraces to Promscale through the OpenTelemetry Collector by configuring it to receive traces in those formats and convert them to OpenTelemetry.Promscale supports the full OpenTelemetry trace schema. It stores and queries all the data, including resources, spans, events, and links with their corresponding attributes. Promscale stores OpenTelemetry traces in TimescaleDB using an optimized database schema and leverages TimescaleDB’s compression capabilities to reduce disk space requirements by up to 10-20x. Theretention period is configurableand is set to 30 days by default.Querying Traces With SQLTo make querying trace data with SQL easier, Promscale exposes database views that make querying spans, events, and links (with all their corresponding attributes) as easy as querying single relational tables.For example, the following query returns the average response time (i.e., latency) in milliseconds for requests to each service in the last five minutes:SELECT
    service_name AS ""service"",
    AVG(duration_ms) as ""latency (ms)""
FROM ps_trace.span s
WHERE 
   start_time > NOW() - INTERVAL '5m'
   AND 
   (parent_span_id IS NULL
   OR
   span_kind = 'SPAN_KIND_SERVER'
   )
GROUP BY 1
ORDER BY 1 ASC;ps_trace.spanis the span view. The conditionparent_span_id IS NULL OR span_kind = 'SPAN_KIND_SERVERensures that only root spans (i.e., those that have no parent span) or spans that represent a service receiving a request from another service (i.e., this is whatkind serverindicates) are selected. Finally, we compute the average of the duration of those spans across the last five minutes for each service.This is the result of visualizing the query above in Grafana with a Table panel:Latency in milliseconds for requests to each service in the last five minutesBecause traces are represented as a hierarchy of spans, the ability to use SQL joins, aggregates, and recursive queries to perform sophisticated analysis on trace data is extremely valuable. We’ll say it again: observability is essentially an analytics problem, and SQL is the most popular tool for analytics.Imagine that we notice our customers are experiencing prolonged response times from a specific API endpoint we offer. When we receive a request to that API endpoint, there are many operations in multiple microservices that are called to fulfill that request. Identifying where the bottleneck is in a distributed environment is not trivial. How could we quickly identify where most of the time is effectively being spent?Note that there are a couple of important considerations to take into account.First, the microservices and operations involved in requests to that API endpoint could also be involved in many other requests. We only want to measure the performance of those service operationswhen they are called as part of a request to that API endpointand not when they are called because of requests to other endpoints.Secondly, we want to look at the effective execution time, that is, the time when a service operation is actually doing some processing and not just waiting for the response from another service operation (within the same service or another service). The graphic below illustrates the problem.In that particular trace, what is the operation that is consuming the most time? Well, if we just look at the duration of each span, the initial span (i.e., the root span) will always take more time since it includes all other spans. But that’s useless because it doesn’t help us identify the bottleneck. To compute the effective execution time of an operation, we need to subtract its duration from the duration of all child spans. In our example, the effective execution time of Operation A is t1 + t2 + t6, which is shorter than t4, which is the effective execution time of Operation C. Therefore, Operation C is the main bottleneck in this request.The root span can be broken down into child spans—the duration of each child span will give us clues on which operation may be causing a bottleneckNote that if we used parallelization or asynchronous operations, we would need to do this differently, but for the sake of simplicity, we’ll assume operations are sequential.To address those two problems, we need to take several steps:Compute the duration of each downstream service and operation only when they are being called as part of a request to the slow API endpointSubtract to each of those the duration of child spansAggregate the results by service and operationTo do this in SQL, we use a recursive query that uses the following syntax:Syntax of a recursive query in SQLThis is how it works.It first runs the initial query, which returns an initial set of results, and then runs the recursive query against that initial set of results to get another set of results. After that, it reruns the recursive query on that set of results and continues doing so until the recursive query returns no results.Then, it takes all the different results from all the individual queries and concatenates them. That’s what the UNION does.And finally, it runs the final query against all those concatenated results—that is what the query returns.The following query is the one we can run to compute the effective execution time of each operation over the last 30 minutes. In this case, the API endpoint we are analyzing is thegenerator.generateoperation (span_nameis the operation’s name) of thegeneratorservice.WITH RECURSIVE x AS
(
    -- initial query
    SELECT
        s.trace_id,
        s.span_id,
        s.parent_span_id,
        s.service_name,
        s.span_name,
        s.duration_ms - coalesce(
        (
            SELECT sum(z.duration_ms)
            FROM ps_trace.span z
            WHERE s.trace_id = z.trace_id
            AND s.span_id = z.parent_span_id
            AND z.start_time > NOW() - INTERVAL '30 minutes'
        ), 0.0) as effective_duration_ms
    FROM ps_trace.span s 
    WHERE s.start_time > NOW() - INTERVAL '30 minutes'
    AND s.service_name = 'generator'
    AND s.span_name = 'generator.generate'

    UNION ALL
    -- recursive query
    SELECT
        s.trace_id,
        s.span_id,
        s.parent_span_id,
        s.service_name,
        s.span_name,
        s.duration_ms - coalesce(
        (
            SELECT sum(z.duration_ms)
            FROM ps_trace.span z
            WHERE s.trace_id = z.trace_id
            AND s.span_id = z.parent_span_id
            AND z.start_time > NOW() - INTERVAL '30 minutes'
        ), 0.0) as effective_duration_ms
    FROM x
    INNER JOIN ps_trace.span s
    ON (x.trace_id = s.trace_id
    AND x.span_id = s.parent_span_id
    AND s.start_time > NOW() - INTERVAL '30 minutes')
    
)
-- final query
SELECT
    service_name,
    span_name,
    sum(effective_duration_ms) as total_exec_time
FROM x
GROUP BY 1, 2
ORDER BY 3 DESCThese are the key things to note in this query:The recursive query syntaxThe initial and the recursive queries compute the effective duration of a span by using a subquery to sum the duration of all child spans. The latter is then subtracted from the duration of the span (coalesce is used to return 0 in case the span has no child spans and the subquery returns NULL)s.duration_ms - coalesce(
(
	SELECT sum(z.duration_ms)
	FROM ps_trace.span z
	WHERE s.trace_id = z.trace_id
	AND s.span_id = z.parent_span_id
	AND z.start_time > NOW() - INTERVAL '30 minutes'
), 0.0) as effective_duration_msThe recursive query uses a join to traverse the downstream spans by selecting the child spans. Since the recursive query runs again and again on the results, applying the recursive query to the previous results ends up processing all downstream spans across all traces that originate in our API endpointINNER JOIN ps_trace.span s
ON (x.trace_id = s.trace_id
AND x.span_id = s.parent_span_id
AND s.start_time > NOW() - INTERVAL '30 minutes')The final query aggregates all the individual execution times by service and operationThis is the result of the query in Grafana using the Pie chart panel. It quickly points out therandom_digitoperation of thedigit serviceas the main bottleneck.Pie chart in Grafana showing the total execution time of each operation. With such a view, we can clearly identify that therandom_digitoperation of thedigitservice is the main bottleneckA Modern APM Experience Integrated Into GrafanaAn overview of the Application Performance Monitoring dashboards for Promscale in Grafana (right-click on ""Open Link in New Tab"" for a better view)Since it is directly integrated into your Grafana instance, you don’t need to set up a new tool or learn a new user interface. Additionally, the Grafana dashboards can be updated, which means you can customize them and extend them to meet your specific needs.The new APM experience within Grafana consists of six dashboards linked to each other:[1] Overview: provides a general overview of the performance across all services to surface the most common problems that could require your attention.[2] Service Details:provides a detailed view of the performance of a service to quickly identify specific problems in that service related to throughput, latency, and errors.[3] Service Map:a real-time automatic map of how your services communicate to identify dependencies and quickly validate their implementation.[4] DownstreamDependencies:a real-time detailed node graph with all the downstream services and operations across all selected service and operation traces. This helps you troubleshoot in detail how downstream services and operations impact the performance of an upstream service.[5] UpstreamDependencies:a real-time detailed node graph with all the upstream services and operations across all selected service and operation traces. This helps you quickly identify the potential blast radius of an issue in a specific service and operation and determine which upstream service and operation are causing problems (like a sudden increase in requests impacting performance) on a downstream service.Note: Some of these dashboards use the Node graph panel, which was introduced in recent versions of Grafana. It’s a beta version, so it may break. It worked for us in Grafana 8.5.1.These dashboards are available onGitHub(filename starts with apm-). Checkour documentationfor details on how to import the dashboards into Grafana.Observability Is an Analytics ProblemOpenTelemetry, a CNCF standard for instrumentation, makes telemetry data easier to collect and universally available. Traces, in particular, hold a treasure of valuable information about how distributed systems behave. We need powerful tools to analyze the data to get the most value from tracing. Observability is essentially an analytics problem.SQL is the lingua franca of analytics. When applied to trace data, it helps you unlock new insights to troubleshoot production issues faster and proactively improve your applications. Get started now withPromscale on Timescale (free 30-day trial, no credit card required)orself-hostfor free.Promscale is an observability backend built on the rock-solid foundation of PostgreSQL and TimescaleDB. With the new support for OpenTelemetry traces and integrations with the visualization tools you use and love, you have full SQL superpowers to solve even the most complex issues in your distributed systems. You can install Promscale for freehere.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/observability-powered-by-sql-understand-your-systems-like-never-before-with-opentelemetry-traces-and-postgresql/
2021-10-28T13:15:09.000Z,Massive Scale for Time-Series Workloads: Introducing Continuous Aggregates for Distributed Hypertables in TimescaleDB 2.5,"Announcing TimescaleDB 2.5Time-series datais everywhere, representing how systems, processes, behaviors, and values change over time. By analyzing data across the time dimension, developers can understand what is happening right now, how that is changing, and why that is changing. Time-series data helps developers solve problems in observability, product and user analytics, fintech and blockchain applications, and the Internet of Things, to name but a few industries.And time-series data is also relentless. Because of the sheer volume and rate of information, time-series data can be complex to store, query, and analyze.This challenge (and need!) is why we builtTimescaleDB, a multi-node, petabyte-scale, completely free relational database for time-series. Multi-node TimescaleDB is a distributed system built with the most demanding time-series workloads in mind, scaling to over 10 million metrics ingester per second, storing petabytes of data and processing queries even faster thanks to better parallelization, yet remains built on the rock-solid foundation and maturity of PostgreSQL.Today, there are nearly three million active TimescaleDB databases running mission-critical time-series workloads across industries. This includes a growing number of multi-node deployments, from startups to major enterprises, such as a Fortune 100 tech company that has deployed two production clusters of multi-node TimescaleDB, each with more than 20 servers, to ingest more than a billion rows of telemetry events per day per cluster.TimescaleDB is built for time-series data, and how we designed its scale-out architecture is no exception. While most distributed database systems use one-dimensional sharding to partition data (e.g., by device, stock ticker, or some other application identifier), multi-node TimescaleDB leverages two-dimensional, fine-grained partitioning for greater flexibility, particularly around time-oriented data lifecycle management. This architecture enables not only distributed operation, but also flexible elasticity and data rebalancing, node heterogeneity, data retention policies, efficient data tiering, and more. See ourannouncement blogfor more on the design of multi-node TimescaleDB.One of our goals for multi-node TimescaleDB is to provide a developer experience that’s very similar to that of single-node TimescaleDB, so that eventually, a developer can start with a small single-node deployment, grow over time, and scale-out automatically when needed. Towards this end, recent releases have added functionality to multi-node TimescaleDB to close this gap, such as distributed compression policies, triggers, and generated columns for distributed operations, distributed restore points, additional distributed query push-down optimizations, and more.Continuous aggregates for distributed hypertablesWith today’s release of TimescaleDB 2.5, we are excited to release a highly-requested new capability:continuous aggregates support for distributed hypertables.Continuous aggregates pre-compute results, so that queries on the pre-computed results can be orders of magnitude faster. But this isn’t just a simple key-value cache. With TimescaleDB continuous aggregates, users can query these pre-computed results with the full power of SQL and a powerful query engine, enabling a user to JOIN results against other relational tables, re-aggregate or rebucket the results (e.g., grouping minutely data into 10 minute buckets, or aggregating across many devices), and more.While continuous aggregates are similar to materialized views in PostgreSQL, unlike materialized views, they are incrementally updated with a built-in refresh policy that makes the aggregates stay up-to-date. TimescaleDB’s continuous aggregates also properly handle late or out-of-order data (with minimal overhead), and support the real-time aggregation of pre-computed and raw data. More on this below.Today’s release of continuous aggregates on distributed hypertables retains the same user experience and API that developers already know with hypertables on single-node TimescaleDB. Now, developers can compute continuous aggregates in a multi-node TimescaleDB cluster, massively speeding up workloads that need to process large volumes of data at the petabyte scale in a distributed environment.It’s just a few commands to create a distributed hypertable and continuous aggregate:-- Create a distributed hypertable partitioned on time and device
CREATE TABLE conditions (
   time TIMESTAMPTZ NOT NULL,
   device_id TEXT NOT NULL,
   temperature DOUBLE PRECISION NULL
);
SELECT create_distributed_hypertable('conditions', 'time', 'device_id');

-- Insert some data into the distributed hypertable
INSERT INTO conditions VALUES
   (now() - interval '15m', 'A001', 70.0),
   (now(), 'A001', 90.1),
   (now(), 'B015', 68.5),
   (now(), 'C183', 69.4),
   (now(), 'E602', 73.8);
   
-- Create a continuous aggregate on the distributed hypertable
CREATE MATERIALIZED VIEW conditions_hourly 
   (hour, device_id, mintemp, maxtemp)
WITH (timescaledb.continuous) AS
   SELECT time_bucket('1hour', time), device_id,
      min(temperature), max(temperature)
   FROM conditions
   GROUP BY time_bucket('1hour', time), device_id;

-- Query the continuous aggregate on the distributed hypertable
SELECT * FROM conditions_hourly
   WHERE hour > now() - interval '7day' AND maxtemp > 90;In addition to continuous aggregates for distributed hypertables, this release also adds PostgreSQL 14 support, new time bucketing functionality for expanded timezone management, and a number of other bug fixes.See the release notes for a complete list of all the changes!Want to learn more about how to use continuous aggregates in your distributed hypertables, together with other TimescaleDB features like data retention policies? Watch this demo byMiranda Auhl, Developer Advocate here at Timescale:PostgreSQL 14 supportWe are very proud to announce that TimescaleDB 2.5 includes support for PostgreSQL 14, which was releasedone month ago. As supporters of the Postgres community, we want our users and customers to benefit from the latest and greatest that Postgres has to offer when using TimescaleDB.See the main improvements introduced on this new version of Postgres.We made the decisionearly in the design of TimescaleDBto build on top of PostgreSQL. We believed then, as we do now, that building on theworld’s fastest-growing databasehas numerous benefits for our users and customers. Among these benefits is the rock-solid dependability that developers have come to expect from a database with 20+ years of production usage behind it, so that all the goodness and innovation happening in PostgreSQL itself also benefits TimescaleDB users. Building on top of PostgreSQL enables us to innovate rapidly with solutions for ingesting and analyzing time-series data while also leveraging (and contributing to) a database with a reputation for stability.And in fact, while we were working on releasing support for PG14, our own development and testing processesuncovered a serious memory corruption bug in PG14for which we proposed a patch to the upstream.Timezone support in next-gen time bucketingAdditionally, TimescaleDB 2.5 introduces new timezone support totime_bucket_ng, enabling developers to bucket and analyze their data according to a specific timezone - which is particularly useful for applications spanning data or devices worldwide.Thistime_bucket_ngfunction serves as theexperimental“nextgeneration” version oftime_bucket, but rewritten to introduce support for non-fixed time intervals (specifically, months and years, which have a variable duration based on which month or leap year). Currently,time_bucketonly supports fixed time intervals (e.g., seconds, minutes, hours, days, weeks), which led to a (surprisingly complex) rewrite intime_bucket_ngfor more flexible time bucketing, including for use in continuous aggregates. We plan to graduate this function out of the experimental schema in future releases.Getting TimescaleDB 2.5TimescaleDB 2.5 is available immediately for Timescale and self-managed deployments, and will be available in the coming weeks on Managed Service for TimescaleDB.If you’re new to TimescaleDB,create a free accountto get started with a fully-managed TimescaleDB instance (with a free 30-day trial, no credit card required 🔥).You can alsovisit our GitHubto learn more (and, as always, ⭐️ are appreciated!), or join ourSlack communityto share your results, ask questions, get advice, and connect with 7K+ other TimescaleDB enthusiasts. And, if you are interested in getting alotmore involved,we are hiring worldwide!Interested in learning more about this new functionality? Read on for a refresher on multi-node TimescaleDB and continuous aggregates and how we built support for continuous aggregates on distributed hypertables.Shoutout to all the engineers who worked on features in TimescaleDB 2.5: Markos Fountoulakis, Mats Kindahl, Aleksander Alekseev, Alexsander Kuzmenkov, Sven Klemm, Gayathri Ayyappan, Erik Nordström, Fabrizio Mello, Dmitry Simonenko, Nikhil Sontakke, Duncan Moore, and the entire team of reviewers and testers!We’d also like to give a special shoutout to all community members who’ve asked for improvements to multi-node TimescaleDB,time_bucket_ng, and PostgreSQL 14 support, all of which informed our choice of functionality introduced today.A refresher on multi-node TimescaleDBMulti-node TimescaleDBenables users to run petabyte-scale workloads across multiple physical TimescaleDB instances, allowing users to scale beyond the limits of a single TimescaleDB (or PostgreSQL) instance. To do this, we introduced the concept of adistributed hypertable.A regularhypertable, one of TimescaleDB’s original innovations, is a virtual table in TimescaleDB that automatically partitions data into many sub-tables (“chunks”) on a single machine, continuously creating new ones as necessary, where each chunk includes data belonging to a specific range of timestamps. Yet, it provides the illusion of a single continuous table across all time (the “hypertable”).Adistributed hypertableis a hypertable that is spread across multiple machines, while still maintaining the illusion (and user experience) of a single continuous table across all time. When using the recommended time-and-space partitioning, each chunk in a distributed hypertable is defined by both a time interval and some subset of the keys belonging to the additional partition key. These chunks are automatically created across the distributed hypertable’s data nodes.The multi-node TimescaleDB architecture consists of an access node (abbreviated as AN), which stores metadata for the distributed hypertable and performs query planning across the cluster, and a set of data nodes (abbreviated as DNs), which store individual (or replicated) chunks of the distributed hypertable, and execute queries against those chunks locally. TimescaleDB remains a single piece of software for operational simplicity; any database can become an access node when a user executes theadd_data_nodecommand, passing in the hostnames of other nodes.A distributed hypertable covering one access node (AN) and three data nodes (DN1, DN2, DN3). A distributed hypertable is partitioned into chunks by both “time” and “space” dimensions, which are spread dynamically across available data nodes.Once multi-node TimescaleDB is set up, creating a distributed hypertable is as simple as creating a regular hypertable, as also shown above:-- Create a distributed hypertable partitioned on time and device
CREATE TABLE conditions (
   time TIMESTAMPTZ NOT NULL,
   device_id TEXT NOT NULL,
   temperature DOUBLE PRECISION NULL
);
SELECT create_distributed_hypertable('conditions', 'time', 'device_id');Whenever data is inserted, the access node routes each data tuple to the proper chunk(s) on one or more of its data nodes (in this example, based on timestamp and device id). And as time progresses, the access node will automatically create new chunks on data nodes (corresponding to new time ranges and space partitions) as data continues to be ingested. This ensures that data is spread across the data nodes evenly.Today's multi-node TimescaleDB supports several capabilities which make it suited for large-volume time-series workloads, including efficient query pushdown,automated compression policies,improved data rebalancing for elasticity andhigh availability, distributed object management (e.g., keeping roles, UDFs, and other objects consistent across nodes), and more, all while striving to provide the same experience and functionality as single-node TimescaleDB.A refresher on continuous aggregates (and why they’re powerful)Users are often interested inaggregatescomputed over data, such as the average sensor reading per device per day, the OHLC (open high low close) values per stock per interval, the maximum CPU utilization per 5 minutes, the number of distinct visitors on each web page per day, and so forth.PostgreSQL has powerful support for computing various forms of aggregates, and TimescaleDB’shyperfunctionsadd even more, including various approximate statistics and approximation/sketching algorithms. But computing the aggregates for large tables can be expensive, simply because you have to read and process all the data belonging to the desired interval.To avoid the constant recomputation each time you want to read the aggregated data, many relational databases, including PostgreSQL, support materialized views. The use of materialized views avoid constant recomputation at query time, but come with three limitations for time-series data:The materialized view is not automatically refreshed, so it is necessary to do a regular refresh of the materialized view as new data comes in.When refreshing the materialized view, all the data is read again to compute the aggregate, even if only a small portion of the data is changed.This recomputation necessitates retaining all the underlying raw data in order to perform the refresh of the materialized view.TimescaleDB’scontinuous aggregatesare designed to overcome these three limitations.Continuous aggregates look and behave like materialized views, but they are incrementally updated with a built-in refresh policy that makes the aggregates stay up-to-date as new data is added. Further, they can efficiently handle late or out-of-order data: if data that is inserted, updated, or deleted actually has an older timestamp, the aggregates corresponding to that older period of time are correspondingly updated. But, through the use of clever internal “invalidation records”, the refresh procedure is careful to only refresh data in the materialized view that actually needs to be updated, which avoids recomputing data that did not change. This smart refresh procedure massively speeds up the incremental maintenance of materialized views, and ensures that the aggregate data is up-to-date.Continuous aggregates similarly are amenable to data retention policies that are often applied in time-series workloads, such that aggregates about old data can be retained in the database, even after the corresponding raw data itself has been dropped.When defining a continuous aggregate policy, you define a refresh window. The continuous aggregate will be updated inside this window only, and only the small portions of data that have been changed will be updated, according to internal invalidation records that track changes to old data. Data outside the refresh window won't be refreshed, enabling data retention policies to delete old raw data while still preserving the aggregates.TimescaleDB’s continuous aggregates also supportreal-time aggregation. With real-time aggregation, when querying a continuous aggregate, the query engine will transparently combine (i) results that are already pre-computed in the materialized view with (ii) results computed at query time over the newest raw data in underlying hypertable. (You can turn this off if desired, but the majority of developers want this behavior by default).Real-time aggregation gives you the best of both worlds: the performance of continuous aggregates and the most up-to-date data for real-time queries, without the performance degradation of querying and aggregating all raw data from scratch. This makes real-time aggregation an ideal fit for many real-time monitoring, dashboarding, and analysis use-cases.Continuous aggregates on distributed hypertablesIn TimescaleDB 2.5, these features from continuous aggregates have been extended to distributed hypertables, and use an identicalSQL CREATEcommand as with regular hypertables.But in a multi-node architecture, the continuous aggregate is stored to optimize query performance. In particular, the underlying incrementally materialized view that’s part of a continuous aggregate is stored on the access node, while the raw data continues to reside across the data nodes. Data is asynchronously materialized from data nodes to the access node via the continuous aggregates refresh policy.Queries on continuous aggregates over distributed hypertables achieve high performance. Rather than querying the raw data distributed on the various data nodes and aggregating it at query time, many or all of the results from a continuous aggregate query are already pre-computed and present on the access node. This turns a distributed query across multiple nodes into a local one, while of course also benefiting from results that have been pre-computed as part of the background materialization process, further minimizing the computation needed at query time.Architecture diagram showing how continuous aggregates are stored on a multi-node TimescaleDB cluster. The continuous aggregate is stored on the access node (AN), while the raw data is distributed between the data nodes (DNs). The raw data is materialized in the background by the continuous aggregate refresh policy.Developers familiar with TimescaleDB might wonder what this design means for real-time aggregates. While the pre-computed results in the continuous aggregate might already reside on the AN, the latest raw data (to be aggregated at query time) still resides across the data nodes. But continuous aggregates, even though now on distributed hypertables, continues to support this transparently: the query engine will push down aggregation to the needed DNs for the latest raw data, if a user’s query to a continuous aggregate includes a recent time range that has yet to be materialized.For example, if the continuous aggregate typically has materialized data up until the last 30 minutes, then queries that only touch data older than 30 minutes only need to involve the aggregate data already materialized on the access node. If a query involves newer data, however – say, “give me the max temperature per hour for the past six hours” – then the query engine combines this pre-aggregated (on the AN) with real-time aggregated data from the last 30 minutes (computed across the DNs).This diagram illustrates what happens when you query a continuous aggregate on a distributed hypertable. If the new query involves new data which arrived since the last refresh of the continuous aggregate, the query engine combines the pre-aggregated data on the access node (AN) with this new not yet materialized data from the data nodes (in this case, DN 3) at query time. Note that this query will be considerably more efficient than querying raw data altogether, as the majority of results are pre-computed already on the access node - and only the latest data not yet materialized is fetched at query time from data node 3.Real-time aggregation remains enabled by default, although it can be disabled anytime (including for individual sessions). Without real-time aggregation, a query against a continuous aggregate will be performed fully against data already pre-computed on the access node.Supporting continuous aggregates across distributed hypertables did introduce some new technical challenges. Because the distributed hypertable is spread across multiple data nodes, each data node needs to maintain its own invalidation log to reflect changes to the distributed hypertable’s chunks that reside locally. Then, the refresh process needs to ensure that the proper regions are recomputed, even though these invalidation logs are distributed, and that materialization results from different data nodes are properly serialized (and avoid deadlocks) on the access node. One key step to processing these invalidation logs and refreshing the continuous aggregate involves executing all operations as distributed transactions within the multi-node cluster (using TimescaleDB’s two-phase commit protocol). This way, each stage remains atomic, as it is when the process takes place on single-node TimescaleDB.In future releases, we plan to add functionality to distribute continuous aggregates themselves across data nodes (e.g., where the materialization hypertable underlying the continuous aggregate is itself distributed across the data nodes). This would enable much larger continuous aggregates than can be supported today (e.g., those that cannot fit on the access node).Get started todayTimescaleDB 2.5 is available today for Timescale andself-managed deploymentsand will be available in the coming weeks on Managed Service for TimescaleDB.If you’re new to Timescale,create a free accountto get started with a fully-managed TimescaleDB instance (with a free 30-day trial, no credit card required 🔥).Join ourSlack communityto share your results, ask questions, get advice, and connect with 7K+ other developers (our co-founders, engineers, and passionate community members are active on all channels).Visit our GitHubto learn more (and, as always, ⭐️ are appreciated!). And, if these are the types of challenges you’d like to help solve,we are hiring!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/massive-scale-for-time-series-workloads-introducing-continuous-aggregates-for-distributed-hypertables-in-timescaledb-2-5/
2022-03-29T13:00:00.000Z,Introducing One-Click Database Forking in Timescale,"[Note: This blog post was originally published in December 2021 and updated in March 2022 to reflect new functionality released as part of #AlwaysBeLaunching Cloud Week🐯 ☁️.]Announcing improved support for database forking in Timescale. Create a fork of your database, with the same or different resource configurations, in just a few clicks. Forks allow you to conveniently create testing and staging environments, safely test major upgrades, downsize your service, give access to production data while fully isolating the production database, and much more. 🔥We’re excited to continue #AlwaysBeLaunchingMOAR Edition—a week full of exciting new features for Timescale, bringing you MOAR features that makeTimescaleeven MOAR worry-free, scalable, and flexible!Today, we’re releasing improved one-click database forking in Timescale, with new functionality that allows you to easily spin up forks— identical copies of your database—but now withdifferentresource configurations (CPU, memory, or disk) than the primary database.One-click database forking on Timescale gives development teams the ability to perform a variety of important tasks in less time and with more flexibility: what previously took a considerable amount of manual work can now be done in one or two clicks.Database forking is useful for dev, test, and staging environments, including testing for performance regression, application changes, or safe database upgrades.  It similarly allows you to easily evaluate the impact of schema, index, or configuration changes outside your production database. That is, forks help grow your confidence with any in-place production changes.In addition to only taking one or two clicks to create, database forking on Timescale is cost-effective and significantly reduces the risk of testing on your own production database. Forked databases are billed at an hourly rate, so you only pay for the time your forked instance was active. This means that you can safely test the impact of PostgreSQL version upgrades, changes in application code, a new database INDEX, and more for just pennies per hour.Moreover, with the new ability to upsize or downsize your database fork, you have greater control over resource consumption. For example, if turning on compression reduces your Timescale storage consumption by 90 %, you can now fork into a service with much smaller configured storage, leading to lower costs. Or because forking is a seamless way to grant access to your production data (but not to the production database) to data science or business intelligence teams without needing a separate ETL job or data pipeline, you can easily create forks with more or less resources depending on the needs of their projects.To fork a service in Timescale, follow these simple steps:Select the database service you’d like to fork.Navigate to the “Operations” tab, and click the “Fork service” button.Choose your configuration. To create a fork with the same configuration as the parent service, click on “Create fork.” To fork with a different configuration as the parent service, click instead on “Advanced options,” which will allow you to select your forked service's compute and disk size.That’s it!You can create forks with the same or different resource configurations in TimescaleAfter the time of forking, your forked service will be a completely independent database.  While it inherits the settings of its parent, any subsequent changes in the forked service won’t be reflected in the parent database (and vice versa). In the Timescale UI, you will see which database service was forked from.All forked databases maintain their identity to their parent service, making it easy to remember which database the service was originally forked from.Database forking is available to all users of Timescale. If you’re new to Timescale,you can create a free account to get started(100 % free for 30 days and no credit card required).Have questions or feedback? Join our community! You can chat with us in ourCommunity Slack, and for in-depth technical questions, you can use theTimescale Forum. Feel free to ask us anything!Keep reading for more information on how forking works under the hood, get some ideas for how to simplify your development workflows, and get insights on our roadmap for forking in Timescale.A huge thank you to all the engineers and designers who worked on this feature.The Many Uses of Database ForkingThe ability to easily fork your database comes in handy for multiple scenarios. Here are some common ones:Testing.If your team needs a common image of your database for running correctness or performance regression tests, we recommend leveraging forks in the following way.  (1) Create a ""golden image"" of your database by creating schemas, loading data, etc., but then (2) pause your service so that you'll only pay for storage rather than compute costs, and you can prevent unwanted modifications. (3) Now create a fork of your paused database service, and (4) run any testing against the running service.  (5) Finally, once your team is finished, delete the fork. You can repeat this process as regularly as you need in your testing environment. And if you ever want to tweak your base service, just resume it against and treat it like a normal database.Moreover, you can test the impact of different workloads on your database and find the optimal resource configuration to deal with them, helping you reduce uncertainty for seasonal events. For example, you might anticipate a high load event on your database but aren't sure exactly what resources you'd need. You can create forks with different resource configurations and stress test against them to see which configuration performs best.Safer database upgrades.When doing a major upgrade in a production database (e.g., upgrading from PostgreSQL 12 to 13), we recommend forking your service first. This way, you can perform the upgrades on the forked service first, ensuring that there are no issues related to this change. Once you’re sure the upgrade was successful, you can be confident that everything will work well when running it on your production service.Downsize your service.Forks also allow you to easily downsize your service to eliminate unnecessary costs. In certain situations, you may find yourself paying for a significant amount of unused storage; perhaps your data size has been reduced considerably after enabling or optimizingTimescaleDB compression, or perhaps you’re seeing less traffic than originally predicted. If that happens to be the case, you can conveniently downsize your service using forks, i.e., (1) you can create a fork of your original service, assigning it less disk size and/or compute, and (2) connect your application to the fork, which now acts as your primary database. Once your old and more expensive service is no longer active, you can delete it. And if your data volume increases, you can scale up your service again by simply navigating to the “Resources” tab and selecting a larger storage plan.Create and refresh staging environments.An important aspect of a good testing procedure is having a staging environment with production data so you can test the quality of your new features in real-world conditions. Through database forking, you can spin up an exact copy of your production data without affecting the actual production service. Also, as your production data changes over time, it is good practice to refresh your staging service, as the conditions may change—to have an easy procedure for forking makes this task painless.Provide access to production data (but not to the production database).Many times, teams of data scientists or business intelligence analysts might want access to production data to query and analyze. Database forking enables you to provide access to production data, without having to provide access to the database itself. This is especially useful for cases where implementing access control via PostgreSQL might be too complicated. Simply fork the database and provide teams access to a copy of that data and a dedicated connection string while your production database continues to operate unaffected.You can also give the forked database a different set of resources than the original database, depending on the needs of other teams it's being shared with and the duration of their projects. For example, you could create a fork with more CPU/RAM resources if your data science or BI team is doing a short-term project with heavy analytical or OLAP queries—this helps those teams move quickly as they won't have to wait as long for queries to execute while leaving your production database and its operations unaffected.Create a snapshot of your data.Keeping database snapshots can be very useful for auditing and reporting, and also for doing potential analysis or forensics after carrying out an important change in your service configuration.Forking: Under the HoodEvery service running in Timescale has a backup that we regularly test. This backup is more than a snapshot of the data directory: powered by the continuous archiving feature of PostgreSQL, it contains all database changes at a given point in time. It can be used to restore a database even when the original volume containing the data directory is unavailable.To fork a service in Timescale, instead of restoring the backup of the parent service in place, it is restored to a new instance (the fork) that becomes a clone of the original one.See our blog post oncontinuous backup/restore validation on Timescalefor more details.What’s Next?At Timescale, we like to move fast(without breaking things) so this is only the beginning for database forking in Timescale. In the near future, we will release the following functionality:Forking to a different region than the parent service.Forking from an arbitrary point in time (PIT)—other than the latest—so you can fork to older states of your database.A programmatic API to automate forking.One-click forking for multi-node services.Timescale fully supports multi-node deployments, but for now, forking is only available for single-node databases.Stay tuned, and let us know if there’s more forking functionality you’d like to see.And may the forks be with you!Get StartedDatabase forking is available to all users of Timescale.Check out our documentationfor more information on forking.If you are new to Timescale:Create an accountto get free access to Timescale for 30 days, with no credit card required—and start forking today!Join the Timescale Community Slackand ask us any questions about time-series data, TimescaleDB, PostgreSQL, and more. Join us: we are 8,000+ and counting!Read ourvision for Timescale: a database cloud for relational and time-series workloads, built on PostgreSQL and architected around our vision of a modern cloud service: easy, scalable, familiar, and flexible.Check out our Getting Started documentation. These articles walk you through the basics: creating your first instance, accessing your database, loading your data, and so on. Get familiar with key TimescaleDB concepts likehypertables and chunks,compression, orcontinuous aggregates. Understanding these key features will allow you to use Timescale to its full potential. ✨And, for those who share our mission of serving developers worldwide 🌏 and want to join our fully remote, global team,we are hiring broadly across many roles!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/introducing-one-click-database-forking-in-timescale-cloud/
2023-05-17T16:41:39.000Z,Announcing the New Timescale,"We used to be a database company with a cloud product. Now we are a cloud company with a database product.Today, we are excited to announce some major changes. These changes represent another step in our journey as we build the next great database company to power the future of computing:Timescale Cloud, our main cloud product, is now just “Timescale,” our primary product.TimescaleDB will continue as our open-source PostgreSQL extension for time series and a key part of the Timescale offering. We will not only continue to improve the TimescaleDB extension but also work with other PostgreSQL extensions to better serve our community and customers.To match our change in identity, we are also evolving our brand with a new logo and color palette.These changes represent an evolution of our business. We used to be a database company with a cloud product. Now we are a cloud company with a database product.This is the new Timescale.Old Timescale logo vs. newAnd this is just the beginning. There is much more to come in the next few months.To see the new Timescale, please visitour new website.To learn more about these changes, please read on.Building a Database Business in the Cloud Era (Version 3)We first launched Timescale over six years ago. In those years, we had a front-row seat to a major shift in developer needs to the cloud. Back then, cloud was an afterthought, just one way to deploy database software.But today, we see the dominance of cloud-first architectures, where developers often choose not between database software products but between database cloud products.In response to that shift, over the past several years, we’ve made bold, industry-leading moves, including:Announcing the Timescale License(2018), a new source-available software license that preserves the openness of open source while allowing us to build a sustainable business. (In particular, while other database companies have relicensed their open-source software to proprietary licenses, we haveneverre-licensed our open-source software.) This was version 1.Going ""all in"" commercially on Cloud(2020), when we made all of our software features free, we decided to monetize solely via our cloud service and added rights to the Timescale License. At a time when other database companies were locking down their products and making them even more closed, we went the other way, opening up our Timescale License by adding the “right-to-repair” and the “right-to-improve” and eliminating the paid enterprise tier and usage limits altogether (thus establishing that all of our software will be available for free). This was version 2.Today’s changes are “version 3”: another step in our journey of serving developers worldwide as they move toward more cloud-first deployments.By continuing to embrace the cloud, we have an opportunity to reinvent the databaseWorking with thousands of companies, we have realized this: the cloud is a tectonic shift in how software infrastructure is distributed and consumed, and we are still in the early stages of this change. By continuing to embrace the cloud, we have an opportunity to reinvent the database.Reinventing the Database in the CloudWhen we started this company, “cloud-native” was already a common buzzword, so we assumed the industry had already figured out how to build cloud-native databases.But as we started listening to our customers and built solutions to their problems, we realized there was still a lot of untapped potential in what a database could offer in the cloud.For example, over a year ago, we began developingdata tiering to S3, a feature launched earlier this year in early access. This unique capability, exclusively available in our cloud product, enables developers to effortlessly scale PostgreSQL tables from disk to object storage, achieving limitless database scalability at a minimal cost.We could have chosen to build this feature in our database software product. But by building it natively in our cloud offering, we were able to deliver a far better user experience for customers in a far shorter amount of time.Based on that experience, we have identified and are now developing more cloud-native features to solve big developer problems. Today’s changes are setting the foundation for that future work.TimescaleCloudand TimescaleDBThe first major change we are making today is renaming our cloud product from “Timescale Cloud” to just “Timescale.”As we started to consider more and more cloud-native database architectures, we realized that the “cloud” suffix felt like an anachronism, a symbol of the past, something redundant. It felt like calling your iPhone a “Smart Phone.” It’s just a phone.Just like most people today have a phone that is “smart,” we believe most developers will soon be consuming their database infrastructure via a “cloud” platform.At the same time, we have a large open-source community of tens of thousands of businesses using our TimescaleDB open-source product. So we will continually develop features in that extension—but only when it provides the best user experience. E.g., improvements to hypertables, compression, and continuous aggregates—some key features that define TimescaleDB—will continue to occur inside the extension.Timescale is more than just a PostgreSQL extension. Timescale is “supercharged PostgreSQL” (or PostgreSQL++): PostgreSQL made easier, faster, and more cost-effectiveSo our PostgreSQL extension will remain “TimescaleDB.” But our main product will be “Timescale,” which will continue to offer TimescaleDB, but also much more, like data tiering to S3.Changing Our StripesLast December, we finalized a completely new brand for Timescale. It was hard to keep this a secret, and today, five months later, we are thrilled to finally share this with you.Timescale is PostgreSQL SuperchargedThis rebrand underscores our evolution as a company. We started with building a PostgreSQL extension for time series, but now Timescale is more than just a PostgreSQL extension. Timescale is “supercharged PostgreSQL” (or PostgreSQL++): PostgreSQL made easier, faster, and more cost-effective. So we wanted to inject a “charged” feeling into the identity system.Electric yellowWe updated our color palette to support our evolved identity.We retained yellow as the primary color but amped it to an “electric” version that feels bolder while using darker background colors to highlight the contrast.The story behind the color: during an especially fun brainstorming session, one of us (Ajay) was reminiscing about playingStreet Fighter IIas a kid at his best friend’s house and how one of the characters “electrified” as their special move. And thus, “Blanka yellow” became the inspiration behind our primary color, “electric yellow.”Along with the yellow, we added a more restrained blue and purple in the palette to give us the flexibility to shift and project a warm, softer image when necessary.The new Timescale color paletteThis color palette is now expressed in our website, cloud console, and docs.The newTimescale homepageTheTimescale DocsThe Services overview page in the rebranded Timescale UIA New Timescale TigerWe have always embraced the “tiger” as the personification of our database: fast and powerful. We now have a new tiger logo, which pays tribute to our brand’s heritage while also evolving it to feel more modern, innovative, and optimistic.The evolution of the Timescale tigerThe new Timescale tiger is forward-facing, signaling an innovative, forward-looking, optimistic company. It is less aggressive and more hopeful. It is also designed to scale up and down more easily.Our ""tiger"" stripes represent the dynamic character of time-series dataThe tiger “stripes” carry through to the entirety of the identity system. Dynamic and twisting, they are inspired by data: ever-changing, alive, and on the move.More to ComeThis is just the beginning. With these changes today, we are laying the groundwork for more awesomeness as we continue to serve developers worldwide and power the future of computing. Stay tuned.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/announcing-the-new-timescale/
2023-11-06T12:50:04.000Z,Introducing Dynamic PostgreSQL: How We Are Evolving the Database to Help You Sleep at Night,"Starting today,you can create Dynamic PostgreSQL databases on Timescale.Dynamic PostgreSQL is the natural evolution of cloud databases, solving the problems of both provisioned databases and serverless databases. It is backed by dynamic compute, a Timescale innovation that instantaneously scales your available compute within a predefined min/max range according to your load. Instead of provisioning for your peak (and paying for it at all times), you can now pick a compute range: your database will operate at base capacity and instantly access up to its peak only when needed. Buy the base, rent the peak.This results in unparalleled price-performance: customers running production workloads will save 10-20 % when migrating from AWS RDS for PostgreSQL and 50-70 % when migrating from AWS Aurora Serverless.You can try out Dynamic PostgreSQL today.We offer a free trial—no credit card required—that gives you full access to the platform for 30 days.To create a Dynamic PostgreSQL service, just select the PostgreSQL option when logging into Timescale:You can now create Time Series services and PostgreSQL services on the Timescale platformYour application is always on, why shouldn’t your database be?Welcome to the future.Problem 1: Developers provision far more compute than they needFor the past several years, ever since we first launched Timescale, we have had a front-row seat to how developers use databases. For example, just in the last few months,we have analyzed over one trillion queriesas part of our Insights product.One thing we have learned is that developers often provision far more compute than they actually need.On the one hand, this makes sense: You never want to worry about your database. Most database workloads are continuous, typically with some variability or burstiness to them. For example, a game that has more usage at night, a business application that has more usage during the day, or a connected home device that has more usage on the weekends than during the week.You never want your database to run out of resources. If your database is at max capacity, that leads to a terrible customer experience (or no customer experience!). So, most developers end up provisioning for the peak, plus a buffer. This results in developers paying for much more compute than they actually need.On the other hand, this seems crazy to us. What other business resource would organizations be okay spending far more than what they need? Wasted compute equals wasted money.Problem 2: Serverless databases fall short for production workloadsSome of you might be asking, “What about serverless databases?”The concept of serverless originated with stateless workloads. After the success of virtual machines in the cloud, where users could stop needing to worry about hardware, they next asked why even worry about running application servers at all? After all, many users just wanted to run functions and only be charged for the time those functions were running. And it’s easy and seamless to spin up functions as needed, almost precisely because they are stateless. Serverless—and function-as-a-service or FaaS—became a hit, with AWS Lambda taking over.Developers then asked themselves, “Why pay for my database when I’m not using it?” The actual question is good: wasted resources are a massive database problem. And the practice of provisioning an AWS RDS database on a specific server instance (say, a db.m6gd.2xlarge) sure doesn’t feel modern or flexible: fixed CPU, fixed memory, fixed local disk. Most of it underutilized most of the time.But this is where things get tricky: databases are very different from Lambda functions.Serverless databases today are wrong for most production workloads for two main reasons:Serverless databases focus on the extremes for scaling up and down, even to zero.Serverless databases introduce much higher pricing to account for the resource “headroom” reserved to serve changing demands (and worse, often with pricing models that are hard to understand or predict).Let’s start by discussing the hot topic of “scale to zero.” The reality is that most production databases don’t need and won’t actually benefit from scaling to zero.Now, there are some use cases where “scale to zero” makes sense. For example, proof-of-concept demos or more hobbyist applications. The ability to occasionally run an ad-hoc query against your dataset (AWS Athena and Google BigQuery make a strong case for a low-cost, serverless cloud data warehouse for very intermittent use). Another suitable use case would be to avoid forgetting to spin down a cloud dev instance once finished—there’s value in “auto pausing” a non-production database (although that requires much simpler functionality than envisioned by serverless).But for your production database and in more operational settings? You don’t want to scale to zero.Scaling to zero means a “cold boot” on restart: empty database shared buffers, empty OS cache, empty catalog caches (in the case of PostgreSQL).(Yes, some serverless databases lower the time it takes to start the database running, but they do so from an empty state. In a relational database like PostgreSQL, it can take minutes (or longer!) to build a warm working set again, especially for larger databases.)The cold start performance hit is even greater as many serverless databases adopt different cloud storage architectures, where the cost and latency of fetching database pages from remote storage into memory is even greater. These overheads again lead to worse performance or force platform providers to compensate by using greater physical resources (e.g., Amazon Aurora databases have twice the memory of RDS), a cost that ultimately gets passed on to users.So, in many scenarios, serverless databases end up with higher and unpredictable pricing.For example, if you compare Aurora Serverless against Amazon RDS, you’ll see that 8 vCPU compute and 500 GB storage on Serverless is 85 % more expensive than RDS ($1,097 vs. $593). And this is using Aurora I/O Optimized and its more predictable storage prices, which launched just six months ago. (Although, even here, we still have to infer its actual compute capacity, as Aurora Serverless prices by confusing opaque “Aurora Capacity Units,” which to our best-informed estimates are 1 ACU = 0.25 vCPU.)Editor’s note:We’ll be publishing a complete benchmark backing those results soon. Stay tuned.Previously, with Aurora Standard, users would also pay for each internal I/O operation, which was nearly impossible to predict or budget. Many serverless databases continue to charge for such reads and writes.  In fact, when we benchmarked the serverless AWS Timestream, we sawcosts that ended up more than 100x higherthan with Timescale due to all of these higher marginal costs.  The unpredictability and variability of costs were the opposite of worry-free.In short, serverless databases are prone to poorer performance, unpredictable bills, and high costs as workloads scale. They are only well suited for intermittent workloads that spin up only occasionally and can tolerate cold starts with their lack of in-memory data caching.The developer dilemmaThis is where we have ended up:Many developers still choose traditional DBaaS services with provisioning for production applications due to their reliable performance, control, and understandability, but hate the waste that arises from the necessity of overprovisioning.Some developers choose serverless databases for their apparent cost savings, flexibility, and ease of use, but hate the performance hit and the unpredictable, obscure pricing (which often result in bills mysteriously higher than a provisioned instance).As developers ourselves, neither of these options is very appealing! There is an opportunity for better.Solution: Introducing Dynamic PostgreSQLThat’s why we developed Dynamic PostgreSQL.Dynamic PostgreSQL consistently supports your baseline and seamlessly scales compute when you need it, up to a defined max. This makes it perfect for the range of continuous workloads you typically see in production settings (whether uniform, variable, or bursty).Dynamic PostgreSQL is 100 % PostgreSQL, with all the benefits of the PostgreSQL community and ecosystem, plus thematurity of Timescale’s database platform. To build Dynamic PostgreSQL, we’ve innovated on how we operate our PostgreSQL infrastructure rather than modifying the internals of PostgreSQL. This gives you access to everything that PostgreSQL—and the Timescale platform—offers, without the fear of running on a forked PostgreSQL query or storage engine.With Dynamic PostgreSQL, you choose a computerange(a minimum and maximum CPU) corresponding to your workload needs. This compute range also comes with effective memory that is equivalent to what most DBaaS services traditionally offer on the “maximum” end of the compute range.The base (minimum) of your CPU range acts exactly like the provisioned DBaaS model: the minimum CPU is at all times dedicated to your service to run your application. As your load increases—either due to your external application’s demand or even due to occasional internal database tasks like incremental backups or table auto-vacuuming—your database can use up to the peak (maximum) of your CPU range with zero delay.How do we achieve zero delay? Dynamic compute works differently than some other serverless or auto-scaling database offerings, so it doesn’t involve the slow scaling (and performance hit) you typically see from remote migrations. Instead, our infrastructure configuration and workload placement algorithms ensure that databases can scale up on their underlying node without restarts or reconfiguration. Your instance always has access to its maximum compute as needed.And the best part is that you only pay for the base plus what you use above it. We call this model of choosing a compute range and scaling between it “buy the base, rent the peak.”For example, if you choose a 4–8 CPU option, you will always have 4 CPUs dedicated to your service and 32 GB of effective memory. This ensures good base performance at all times. When your load increases, your application can use up to 8 CPUs instantaneously as it needs—metered and billed on a fractional CPU basis—and never more than 8 CPUs if this is your max limit.The dynamic model allows you to “size” your database more cost-effectively and worry-free. You can choose a compute range where your standard demand fits the minimum, yet you can grow or spike up to the peak (maximum) as needed. This maximum creates an inherent cap on any usage above your base compute, leading to an easy-to-understand cost ceiling. Further, we charge the same rate per (fractional) CPU-hour for both your base and any metered usage over this: there is no upcharge for using above your base, and therefore no price penalty for scaling.Finally, if you realize you provisioned a size range that is too low or too high, you can easily adjust your compute range to a size that better suits your application’s needs.Engineered to save you moneyWe currently offer five different compute ranges based on your workload size, with corresponding effective memory you receive for the range regardless of your instantaneous usage.Dynamic PostgreSQL also usesTimescale’s usage-based storage, where you only pay for the volume of data stored (in GB-hours), not for a provisioned disk size. No worrying about wasting money with an over-provisioned disk or similarly worrying that you’ll run out of disk space. Timescale’s dynamic cloud infrastructure ensures you have sufficient storage capacity, when you need it, and that you only pay for what you use.We have developed Dynamic PostgreSQL intentionally to save you money. Customers running production workloads typically save 10-20 % when migrating from AWS RDS for PostgreSQL and 50-70 % when migrating from AWS Aurora Serverless.At the end of the month, your bill consists of two simple, easy-to-understand metrics: (1) your compute costs, billed as your hourly base compute plus any fractional CPU usage above it but no more than your peak; and (2) your storage costs, billed as data consumption in GB-hours. There are no new metrics or derived units to measure or understand.Just pay for what you use. Zero extra costs or hidden fees.Compute: predictable, based on a defined rangeStorage: only pay for what you storeNo wasted resources. No overpaying. No losing sleep at night. A bill you can explain to your boss.Try it todayYou can try out Dynamic PostgreSQL today!Timescale offers a free trial—no credit card required—that gives you full access to the platform for 30 days. To create a Dynamic PostgreSQL service, just select the PostgreSQL option when logging into Timescale:You can now create Time Series services and PostgreSQL services on the Timescale platformThe platform now offers two service types to serve the specific needs of your databases:Time-series servicesare engineered to boost query speed and scalability for your most demanding workloads, offering key Timescale features such as hypertables, columnar compression, continuous aggregates, and tiered storage.  Use them to host your sensor data, energy metrics, financial data, events, and other data-intensive workload.PostgreSQL servicesare dynamic Postgres services optimized for cost-efficiency and ease of use. Use them for your relational-only databases, e.g., business records.Once you select “PostgreSQL,” configuring your Dynamic PostgreSQL service is super simple. Select your region, your dynamic compute range, and your high availability and connection pooling options— boom! 💥 You now have a Dynamic PostgreSQL database ready to use in production.Select the dynamic compute option that best fits your workload. And if you’re not sure, no problem—you can change it anytime.If you have any questions,reach out to us. We’d love to hear your feedback and to help you with your PostgreSQL use case (time series or not)!This is just the beginning. We’re in the middle of three consecutive launch weeks, and this is just the start of Week 2: Dynamic Infra Week. Stay tuned for more this week, this month, this year, and the many years to come. 🙂Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/introducing-dynamic-postgresql/
2022-03-30T13:15:09.000Z,Using pg_stat_statements to Optimize Queries,"pg_stat_statementsallows you to quickly identify problematic or slow Postgres queries, providing instant visibility into your database performance. Today, we're announcing that we've enabledpg_stat_statementsby default in all Timescale services. This is part of our #AlwaysBeLaunching Cloud Week with MOAR features!🐯☁️PostgreSQL is one of the fastest-growing databases in terms of usage and community size, being backed by many dedicated developers and supported by a broad ecosystem of tooling, connectors, libraries, and visualization applications. PostgreSQL is also extensible: usingPostgreSQL extensions, users can add extra functionality to PostgreSQL’s core.  Indeed, TimescaleDB itself ispackaged as a PostgreSQL extension, which also plays nicely with the broad set of other PostgreSQL extensions, as we’ll see today.Today, we’re excited to share thatpg_stat_statements, one of the most popular and widely used PostgreSQL extensions, is now enabled by default in all Timescale services. If you’re new to Timescale,start a free trial(100 % free for 30 days, no credit card required).What is pg_stat_statements?pg_stat_statementsis a PostgreSQL extension that records information about your running queries. Identifying performance bottlenecks in your database can often feel like a cat-and-mouse game. Quickly written queries, index changes, or complicated ORM query generators can (and often do) negatively impact your database and application performance.How to use pg_stat_statementsAs we will show you in this post,pg_stat_statementsis an invaluable tool to help you identify which queries are performing slowly and poorly and why. For example, you can querypg_stat_statementsto know how many times a query has been called, the query execution time, the hit cache ratio for a query (how much data was available in memory vs. on disk to satisfy your query),  and other helpful statistics such as the standard deviation of a query execution time.Keep reading to learn how to querypg_stat_statementsto identify PostgreSQL slow queries and other performance bottlenecks in your Timescale database.A huge thank you to Lukas Bernert, Monae Payne, and Charis Lam for taking care of all things pg_stat_statements in Timescale.How to Querypg_stat_statementsin TimescaleQuerying statistics data for your Timescale database from thepg_stat_statementsview is straightforward once you're connected to the database.SELECT * FROM pg_stat_statements;

userid|dbid |queryid             |query                         
------+-----+--------------------+------------------------------
 16422|16434| 8157083652167883764|SELECT pg_size_pretty(total_by
    10|13445|                    |<insufficient privilege>      
 16422|16434|-5803236267637064108|SELECT game, author_handle, gu
 16422|16434|-8694415320949103613|SELECT c.oid,c.*,d.description
    10|16434|                    |<insufficient privilege>      
    10|13445|                    |<insufficient privilege>   
 ...  |...  |...                 |...Queries that thetsdbadminuser does not have access to will hide query text and identifierThe view returns many columns of data (more than 30!), but if you look at the results above, one value immediately sticks out:<insufficient privilege>.pg_stat_statementscollects data on all databases and users, which presents a security challenge if any user is allowed to query performance data. Therefore, although any user can query data from the views, only superusers and those specifically granted thepg_read_all_statspermission can see all user-level details, including thequeryidandquerytext.This includes thetsdbadminuser, which is created by default for all Timescale services. Although this user owns the database and has the most privileges, it is not a superuser account and cannot see the details of all other queries within the service cluster.Therefore, it's best to filterpg_stat_statementsdata byuseridfor any queries you want to perform.-- current_user will provide the rolname of the authenticated user
SELECT * FROM pg_stat_statements pss
	JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user;


userid|dbid |queryid             |query                         
------+-----+--------------------+------------------------------
 16422|16434| 8157083652167883764|SELECT pg_size_pretty(total_by
 16422|16434|-5803236267637064108|SELECT game, author_handle, gu
 16422|16434|-8694415320949103613|SELECT c.oid,c.*,d.description
 ...  |...  |...                 |...Queries for only thetsdbadminuser, showing all details and statisticsWhen you add the filter, only data that you have access to is displayed. If you have created additional accounts in your service for specific applications, you could also filter to those accounts.To make the rest of our example queries easier to work with, we recommend that you use this base query with a common table expression (CTE). This query form will return the same data but make the rest of the query a little easier to write.-- current_user will provide the rolname of the authenticated user
WITH statements AS (
SELECT * FROM pg_stat_statements pss
		JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user
)
SELECT * FROM statements;

userid|dbid |queryid             |query                         
------+-----+--------------------+------------------------------
 16422|16434| 8157083652167883764|SELECT pg_size_pretty(total_by
 16422|16434|-5803236267637064108|SELECT game, author_handle, gu
 16422|16434|-8694415320949103613|SELECT c.oid,c.*,d.description
 ...  |...  |...                 |...Query that shows the same results as before, but this time with the base query in a CTE for more concise queries laterNow that we know how to query only the data we have access to, let's review a few of the columns that will be the most useful for spotting potential problems with your queries.calls: the number of times this query has been called.total_exec_time: the total time spent executing the query, in milliseconds.rows: the total number of rows retrieved by this query.shared_blks_hit: the number of blocks already cached when read for the query.shared_blks_read: the number of blocks that had to be read from the disk to satisfy all calls for this query form.Three quick reminders about the data columns above:All values are cumulative since the last time the service was started, or a superuser manually resets the values.All values are for the same query form after parameterizing the query and based on the resulting hashedqueryid.The current configuration for Timescale services does not track query planning statistics because of the small added overhead. We may allow this through user configuration in the future.Using these columns of data, let's look at a few common queries that can help you narrow in on the problematic queries.Long-Running PostgreSQL QueriesOne of the quickest ways to find slow Postgres queries that merit your attention is to look at each query’s average total time. This is not a time-weighted average since the data is cumulative, but it still helps frame a relevant context for where to start.Adjust thecallsvalue to fit your specific application needs. Querying for higher (or lower) total number of calls can help you identify queries that aren't run often but are very expensive or queries that are run much more often than you expect and take longer to run than they should.-- query the 10 longest running queries with more than 500 calls
WITH statements AS (
SELECT * FROM pg_stat_statements pss
		JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user
)
SELECT calls, 
	mean_exec_time, 
	query
FROM statements
WHERE calls > 500
AND shared_blks_hit > 0
ORDER BY mean_exec_time DESC
LIMIT 10;


calls|mean_exec_time |total_exec_time | query
-----+---------------+----------------+-----------
 2094|        346.93 |      726479.51 | SELECT time FROM nft_sales ORDER BY time ASC LIMIT $1 |
 3993|         5.728 |       22873.52 | CREATE TEMPORARY TABLE temp_table ... |
 3141|          4.79 |       15051.06 | SELECT name, setting FROM pg_settings WHERE ... |
60725|          3.64 |      221240.88 | CREATE TEMPORARY TABLE temp_table ... |   
  801|          1.33 |        1070.61 | SELECT pp.oid, pp.* FROM pg_catalog.pg_proc p  ...|
 ... |...            |...                 |Queries that take the most time, on average, to executeThis sample database we're using for these queries is based on theNFT starter kit, which allows you to ingest data on a schedule from the OpenSea API and query NFT sales data. As part of the normal process, you can see that aTEMPORARY TABLEis created to ingest new data and update existing records as part of a lightweight extract-transform-load process.That query has been called 60,725 times since this service started and has taken around 4.5 minutes of total execution time to create the table. By contrast, the first query shown takes the longest, on average, to execute—around 350 milliseconds each time. It retrieves the oldest timestamp in thenft_salestable and has used more than 12 minutes of execution time since the server was started.From a work perspective, finding a way to improve the performance of the first query will have a more significant impact on the overall server workload.Hit Cache RatioLike nearly everything in computing, databases tend to perform best when data can be queried in memory rather than going to external disk storage. If PostgreSQL has to retrieve data from storage to satisfy a query, it will typically be slower than if all of the needed data was already loaded into the reserved memory space of PostgreSQL. We can measure how often a query has to do this through a value known as Hit Cache Ratio.Hit Cache Ratio is a measurement of how often the data needed to satisfy a query was available in memory. A higher percentage means that the data was already available and it didn't have to be read from disk, while a lower value can be an indication that there is memory pressure on the server and isn't able to keep up with the current workload.If PostgreSQL has to constantly read data from disk to satisfy the same query, it means that other operations and data are taking precedence and ""pushing"" the data your query needs back out to disk each time.This is a common scenario for time-series workloads because newer data is written to memory first, and if there isn't enough free buffer space, data that is used less will be evicted. If your application queries a lot of historical data, older hypertable chunks might not be loaded into memory and ready to quickly serve the query.A good place to start is with queries that run often and have a Hit Cache Ratio of less than 98 %. Do these queries tend to pull data from long periods of time? If so, that could be an indication that there's not enough RAM to efficiently store this data long enough before it is evicted for newer data.Depending on the application query pattern, you could improve Hit Cache Ratio by increasing server resources, consider index tuning to reduce table storage, or useTimescaleDB compressionon older chunks that are queried regularly.-- query the 10 longest running queries
WITH statements AS (
SELECT * FROM pg_stat_statements pss
		JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user
)
SELECT calls, 
	shared_blks_hit,
	shared_blks_read,
	shared_blks_hit/(shared_blks_hit+shared_blks_read)::NUMERIC*100 hit_cache_ratio,
	query
FROM statements
WHERE calls > 500
AND shared_blks_hit > 0
ORDER BY calls DESC, hit_cache_ratio ASC
LIMIT 10;


calls | shared_blks_hit | shared_blks_read | hit_cache_ratio |query
------+-----------------+------------------+-----------------+--------------
  118|            441126|                 0|           100.00| SELECT bucket, slug, volume AS ""volume (count)"", volume_eth...
  261|          62006272|             22678|            99.96| SELECT slug FROM streamlit_collections_daily cagg...¶        I
 2094|         107188031|           7148105|            93.75| SELECT time FROM nft_sales ORDER BY time ASC LIMIT $1...      
  152|          41733229|                 1|            99.99| SELECT slug FROM streamlit_collections_daily cagg...¶        I
  154|          36846841|             32338|            99.91| SELECT a.img_url, a.name, MAX(s.total_price) AS price, time...

 ... |...               |...               | ...             | ...The query that shows the Hit Cache Ratio of each query, including the number of buffers that were ready from disk or memory to satisfy the queryThis sample database isn't very active, so the overall query counts are not very high compared to what a traditional application would probably show. In our example data above, a query called more than 500 times is a ""frequently used query.""We can see above that one of the most expensive queries also happens to have the lowest Hit Cache Ratio of 93.75 %. This means that roughly 6 % of the time, PostgreSQL has to retrieve data from disk to satisfy the query. While that might not seem like a lot, your most frequently called queries should have a ratio of 99 % or more in most cases.If you look closely, notice that this is the same query that stood out in our first example that showed how to find long-running queries. It's quickly becoming apparent that we can probably tune this query in some way to perform better. As it stands now, it's the slowest query per call, and it consistently has to read some data from disk rather than from memory.Queries With High Standard DeviationFor a final example, let's look at another way to judge which queries often have the greatest opportunity for improvement using the standard deviation of a query execution time.Finding the slowest queries is a good place to start. However, as discussed in the blog postWhat Time-Weighted Averages Are and Why You Should Care, averages are only part of the story. Althoughpg_stat_statementsdoesn't provide a method for tracking time-weighted averages, it does track the standard deviation of all calls and execution time.How can this be helpful?Standard deviation is a method of assessing how widely the time each query execution takes compared to the overall mean. If the standard deviation value is small, then queries all take a similar amount of time to execute. If the standard deviation value is large, this indicates that the execution time of the query varies significantly from request to request.Determining how good or bad the standard deviation is for a particular query requires more data than just the mean and standard deviation values. To make the most sense of these numbers, we at least need to add the minimum and maximum execution times to the query. By doing this, we can start to form a mental model for the overall span execution times that the query takes.In the example result below, we're only showing the data for one query to make it easier to read, the sameORDER BY time LIMIT 1query we've seen in our previous example output.-- query the 10 longest running queries
WITH statements AS (
SELECT * FROM pg_stat_statements pss
		JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user
)
SELECT calls, 
	min_exec_time,
	max_exec_time, 
	mean_exec_time,
	stddev_exec_time,
	(stddev_exec_time/mean_exec_time) AS coeff_of_variance,
	query
FROM statements
WHERE calls > 500
AND shared_blks_hit > 0
ORDER BY mean_exec_time DESC
LIMIT 10;


Name              |Value                                                |
------------------+-----------------------------------------------------+
calls             |2094                                                 |
min_exec_time     |0.060303                                             |
max_exec_time     |1468.401726                                          |
mean_exec_time    |346.9338636657108                                    |
stddev_exec_time  |212.3896857655582                                    |
coeff_of_variance |0.612190702635494                                    |
query             |SELECT time FROM nft_sales ORDER BY time ASC LIMIT $1|Queries showing the min, max, mean, and standard deviation of each queryIn this case, we can extrapolate a few things from these statistics:For our application, this query is called frequently (remember, more than 500 calls is a lot for this sample database).If we look at the full range of execution time in conjunction with the mean, we see that the mean is not centered. This could imply that there are execution time outliers or that the data is skewed. Both are good reasons to investigate this query’s execution times further.Additionally, if we look at the coefficient of variation column, which is the ratio between the standard deviation and the mean (also called the coefficient of variation), we get 0.612 which is fairly high. In general, if this ratio is above 0.3, then the variation of your data is quite large. Since we find the data is quite varied, it seems to imply that instead of a few outliers skewing the mean, there are a number of execution times taking longer than they should. This provides further confirmation that the execution time for this query should be investigated further.When I examine the output of these three queries together, this specificORDER BY time LIMIT 1query seems to stick out. It's slower per call than most other queries, it often requires the database to retrieve data from disk, and the execution times seem to vary dramatically over time.As long as I understood where this query was used and how the application could be impacted, I would certainly put this ""first point"" query on my list of things to improve.Speed Up Your PostgreSQL QueriesThepg_stat_statementsextension is an invaluable monitoring tool, especially when you understand how statistical data can be used in the database and application context.For example, an expensive query called a few times a day or month might not be worth the effort to tune right now. Instead, a moderately slow query called hundreds of times an hour (or more) will probably better use your query tuning effort.If you want to learn how to store metrics snapshots regularly and move from static, cumulative information to time-series data for more efficient database monitoring, check out the blog postPoint-in-Time PostgreSQL Database and Query Monitoring With pg_stat_statements.pg_stat_statementsis automatically enabled in all Timescale services. If you’re not a user yet,you can try out Timescale for free(no credit card required) to get access to a modern cloud-native database platform withTimescaleDB's top performance, one-clickdatabase replication,forking, andVPC peering.One last thing: to cap off this #AlwaysBeLaunching Cloud Week, we hosted our secondTimescale Community Day.Check out our talks (and demos!)about all things time-series data and databases!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/using-pg-stat-statements-to-optimize-queries/
2023-09-28T13:00:32.000Z,A Python Library for Using PostgreSQL as a Vector Database in AI Applications,"Introducing the Timescale Vector Python client library: a new library for storing, indexing, and querying vector embeddings in PostgreSQL. Easily store millions of embeddings using PostgreSQL as a vector database. Complete with optimized schema, batch ingestion, hybrid search, and time-based vector search. Learn more about its key features. And then take it for a spin:try Timescale Vector today, free for 90 days.Python is the lingua franca of AI. And today, it gets even better for building AI applications with PostgreSQL as a vector database. Introducing the Timescale Vector Python client library, which enables Python developers to easily store, index, and query millions of vector embeddings using PostgreSQL.The Python client library is the simplest way to integrate Timescale Vector’sbest-in-class similarity search and hybrid search performanceinto your generative AI application.Here’s an overview of how the Timescale Vector Python client makes it easier than ever to build AI applications with PostgreSQL:Optimized schema for vectors and metadataPerformant batch ingestion of vectorsCreate Timescale Vector (DiskANN), HNSW (Hierarchical Navigable Small Worlds), and IVFFlat (Inverted File Flat) indexes in one line of codeSemantic search and hybrid searchANN search with time-based filtering of vectorsA foundation for Retrieval Augmented Generation (RAG) with time-based context retrievalIn the remainder of this post, we’ll delve into each of these points with code examples!How to Access the Timescale Vector Python LibraryTo get started with the Timescale Vector python client,sign upto the Timescale cloud PostgreSQL platform, create a new database, and then run the following in your terminal:pip install timescale_vectorThen follow thisup and running with Timescale Vector tutorial(be sure to download the.envfile with your database credentials, you’ll need it to follow the tutorial).Use the Timescale Vector Python library with a cloud PostgreSQL database, free for 90 days.Three-month free trial for Timescale Vector:To make it easy to test and develop your applications with Timescale Vector, we’re giving new Timescale customers a 90-day extended trial. You won’t be charged for any cloud PostgreSQL databases you spin up during your trial period.Try Timescale Vector for free.Special early access pricing:Existing Timescale customers can use Timescale Vector for free during the early access period.Optimized PostgreSQL Schema for Storing Vectors and MetadataTimescale Vector Python client creates an optimized schema to efficiently store vector embeddings and associated metadata for fast search and retrieval. All you need to create a table is a Timescale service URL, your table name, and the dimension of the vectors you want to store.# Table information
TABLE_NAME = ""company_documents""
EMBEDDING_DIMENSIONS = 1536

# Create client object
vec = client.Async(TIMESCALE_SERVICE_URL, 
                   TABLE_NAME,  
                   EMBEDDING_DIMENSIONS)

# create the table and the library handles the schema!
await vec.create_tables()Thecreate_tables()function will create a table with the following schema:id | metadata | contents | embeddingidis the UUID that uniquely identifies each vector.metadatais a JSONB column that stores the metadata associated with each vector.contentsis the text column that stores the content we want vectorized.embeddingis the vector column that stores the vector embedding representation of the content.Performant Batch Ingestion of Vectors With PostgreSQLMost Generative AI applications require inserting tens of thousands of records (embeddings plus metadata) into a table at a time. Timescale Vector makes it easy to batch ingest these records without extra configuration using the.upsert()method:# batch upsert vectors into table
await vec.upsert(records)Create Timescale Vector (DiskANN), HNSW, and IVFFlat ANN Indexes in One Line of CodeWith a single line of code, you can create indices on your vectors to speed up similarity search on millions of embeddings.The Timescale Vector Python library supports thetimescale-vectorindex inspired by the DiskANN algorithm, which achieves3x search speed vs. specialized vector database Weaviate, and between 40% to 1,590% performance improvement over pgvectorwhen performing ANN searches on one million OpenAI embeddings.Timescale Vector’s new index outperforms specialized vector database Weaviate by 243% and all existing PostgreSQL index types when performing approximate nearest neighbor searches at 99% recall on one million OpenAI vector embeddingsYou can create atimescale vector(DiskANN) index in a single line of code:# Create a timescale vector (DiskANN) search index on the embedding column
await vec.create_embedding_index(client.TimescaleVectorIndex())What’s more, the library also supports pgvector’s HNSW and IVFFlat indexing algorithms, along with smart defaults for all three index types. Advanced users can, of course, specify index parameters when creating an index via the index creation method arguments.# Create HNSW search index on the embedding column
await vec.create_embedding_index(client.HNSWIndex())

# Create IVFFLAT search index on the embedding column
await vec.create_embedding_index(client.IvfflatIndex())Similarity Search and Hybrid Vector Search in PostgreSQLThe Timescale Vector Python library provides a method for easy similarity search.As a refresher, similarity search is where we find the vectors most similar in meaning to our query vector—more similar vectors are closer to each other, while less similar vectors are further away in the N-dimensional embedding space. Without indexes, this will default to performing exact nearest neighbor (KNN) search, but with the indexes discussed above enabled, you’ll perform approximate nearest neighbor (ANN) search.# define search query and query_embedding
query_string = ""What's new with Project X""
query_embedding = get_embeddings(query_string)

# search table for similar vectors to query_embedding
records = await vec.search(query_embedding)In addition to simple similarity search (without metadata filters), the Timescale Vector Python library makes it simple to perform hybrid search on your vectors and metadata, where you not only query by vector similarity but by an additional metadata filter or LIMIT:Filters can be specified as a dictionary where all fields and their values are matched exactly. You can also specify a list of dictionaries that uses OR semantics such that a row is returned if it matches any of the dictionaries.We also support using more advanced metadata filters using Predicates. (Seeour documentationfor more details.)Our optimized schema design creates a GIN index on the metadata, allowing optimized searches for many metadata queries.Similarity Search With Time FiltersTimescale Vector optimizes time-based vector search queries, leveraging the automatic time-based partitioning and indexing ofTimescale’s hypertables.Time-based filtering is useful to efficiently find recent embeddings, constrain vector search by a time range or document age, and store and retrieve large language model (LLM) responses and chat history with ease. Time-based semantic search also enables you to use RAG with time-based context retrieval to give users more useful LLM responses.You can use efficient time-based similarity search via the Timescale Vector Python library by first ensuring to create your client with thetime_partition_intervalargument set to the time range you want your data partitioned by as follows:# Table information
TABLE_NAME = ""commit_history""
EMBEDDING_DIMENSIONS = 1536

# Partition interval
TIME_PARTITION_INTERVAL = timedelta(days=7)

# Create client object
vec = client.Async(TIMESCALE_SERVICE_URL, 
                   TABLE_NAME,  
                   EMBEDDING_DIMENSIONS, 
                   time_partition_interval=TIME_PARTITION_INTERVAL)

# create table
await vec.create_tables()In the code block above, we set thetime_partition_intervalargument in the client creation function to enable automatic time-based partitioning of the table. This will partition the table into time-based chunks and create indexes on the time-based chunks to speed up time-based queries.Each partition will consist of data for the specified length of time. We use seven (7) days for simplicity, but you can pick whatever value makes sense for your use case. For example, if you query recent vectors frequently, you might want to use a smaller time delta like one (1) day, or if you query vectors over a decade-long time period then you might want to use a larger time delta like six (6) months or one (1) year.Once we’ve created the table with time partitioning enabled, we can perform time-based similarity searches as follows:# Time filter variables for query
# Start date = 1 August 2023, 22:10:35
start_date = datetime(2023, 8, 1, 22, 10, 35)
# End date = 30 August 2023, 22:10:35
end_date = datetime(2023, 8, 30, 22, 10, 35)

# Similarity search with time filter
records_time_filtered = await vec.search(query_embedding,limit=3, 
                        uuid_time_filter=client.UUIDTimeRange(start_date, end_date))This will ensure our similarity search only returns vectors that have times between thestart_dateandend_date.Here’s some intuition for why Timescale Vector’s time-based partitioning speeds up ANN queries with time-based filters:Timescale Vector partitions the data by time and creates ANN indexes on each partition individually. Then, during search, we perform a three-step process:Step 1: filter our partitions that don’t match the time predicate.Step 2: perform the similarity search on all matching partitions.Step 3: combine all the results from each partition in step 2, rerank, and filter out results by time.Timescale Vector leveragesTimescaleDB’s hypertables, which automatically partition vectors and associated metadata by a timestamp. This enables efficient querying on vectors by both similarity to a query vector and time, as partitions not in the time window of the query are ignored, making the search a lot more efficient by filtering out whole swaths of data in one go.A note on preparing data for time-based partitioningTimescale Vector uses the DateTime portion of a UUID v1 to determine which partition a given row should be placed in.If you want the current date and time associated with your vectors, you can create a newUUID v1for each record that you want to insert:# Table information
TABLE_NAME = ""commit_history""
EMBEDDING_DIMENSIONS = 1536

# Partition interval
TIME_PARTITION_INTERVAL = timedelta(days=7)

# Create client object
vec = client.Async(TIMESCALE_SERVICE_URL, 
                   TABLE_NAME,  
                   EMBEDDING_DIMENSIONS, 
                   time_partition_interval=TIME_PARTITION_INTERVAL)

# create table
await vec.create_tables()If you want a date or time in the past to be associated with your vectors, you can use our handyuuid_from_time()function to generate auuid v1from a Pythondatetimeobject, and then use that as youridfor your vector when you insert it into the PostgreSQL database:# Time filter variables for query
# Start date = 1 August 2023, 22:10:35
start_date = datetime(2023, 8, 1, 22, 10, 35)
# End date = 30 August 2023, 22:10:35
end_date = datetime(2023, 8, 30, 22, 10, 35)

# Similarity search with time filter
records_time_filtered = await vec.search(query_embedding,limit=3, 
                        uuid_time_filter=client.UUIDTimeRange(start_date, end_date))For example, in thistutorial for Timescale Vector, we extract the dates from our metadata and turn them intoUUID v1s, which we then use as theidpart of our record when we ingest into the PostgreSQL table:import uuid
id = uuid.uuid1()
await vec.upsert([(id, {""key"": ""val""}, ""the brown fox"", [1.0, 1.2])])Retrieval Augmented Generation With Time-Based Context RetrievalLet’s put everything together and look at a simplified example of how you can use the Timescale Vector Python library to power retrieval augmented generation where the context retrieved is constrained to a given time range.Generation:In the example below, we defineget_completion_from_messages(), which makes a call to an LLM and returns a completion response for a given prompt.Time-based context retrieval:We defineget_top_most_similar_docs(), which takes a given query embedding and returns the top five most similar rows to that embedding in our table that have a time associated with them betweenstart_dateandend_date.Finally, we put it all together inprocess_user_message, which takes auser_input, like a question, as well as a start and end date, and returns a retrieval augmented response from the LLM using the time-based context retrieved from the records in our table.# Make an LLM call and get completion for a given set of messages
def get_completion_from_messages(messages, model=""gpt-4-0613"", temperature=0, max_tokens=1000):
   response = openai.ChatCompletion.create(
       model=model,
       messages=messages,
       temperature=temperature,
       max_tokens=max_tokens,
   )
   return response.choices[0].message[""content""]

# Get top 3 most similar document sections within a time range
def get_top_similar_docs(query_embedding, start_date, end_date):
   # Get the top most similar documents within the time range
   top_docs = await vec.search(query_embedding,limit=3, uuid_time_filter=client.UUIDTimeRange(start_date, end_date))
   return top_docs

#construct context string from relevant docs array
def process_user_message(user_input, all_messages, start_date, end_date):

   #Get documents related to the user input
   related_docs = get_top_similar_docs(get_embeddings(user_input), start_date, end_date)

   messages = [
       {""role"": ""system"", ""content"": system_message},
       {""role"": ""user"", ""content"": f""{user_input}""},
       {""role"": ""assistant"", ""content"": f""Relevant information: \n {related_docs[0]} \n {related_docs[1]} \n {related_docs[2]}""}
       ]

   final_response = get_completion_from_messages(all_messages + messages)This is a simple example of a powerful concept—using time-based context retrieval in your RAG applications can help provide more relevant answers to your users. This time-based context retrieval can be helpful to any dataset with a natural language and time component.Timescale Vector uniquely enables this thanks to its efficient time-based similarity search capabilities, and taking advantage of it in your Python applications is easy thanks to the Timescale Vector Python client library.Resources and Next StepsNow that you’ve learned the foundational concepts of Timecale Vector’s Python library,install itfor your next project that uses LLMs in Python:pip install timescale_vectorAnd then continue your learning journey with the following tutorials and guides:Python client library documentation: For details on how to use the library.Timescale Vector Python tutorial:Analyze a git log dataset with Timescale Vector. You’ll learn how to ingest and query vectors and metadata and also perform semantic search with time filtering.Using Timescale Vector with LangChain:Learn how to use Timescale Vector in LangChain, the popular LLM development framework. You’ll cover similarity search, RAG with time-based context retrieval, and self-querying.Using Timescale Vector with LlamaIndex:Learn how to use Timescale Vector in LlamaIndex, the popular LLM data framework. You’ll cover custom node creation, time-based context retrieval, and performing RAG with Timescale Vector as a LlamaIndex Retriever and Query Engine.Timescale Vector explainer blog and performance benchmarks: Learn more about Timescale Vector, how it works, and how it compares to specialized vector databases and existing PostgreSQL ANN indexes.And a reminder:Three-month free trial for Timescale Vector:To make it easy to test and develop your applications with Timescale Vector, we’re giving new Timescale customers a 90-day extended trial. You won’t be charged for any cloud PostgreSQL databases you spin up during your trial period.Try Timescale Vector for free.Special early access pricing:Existing Timescale customers can use Timescale Vector for free during the early access period.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/a-python-library-for-using-postgresql-as-a-vector-database-in-ai-applications/
2023-07-12T13:00:15.000Z,How to Build LLM Applications With pgvector Vector Store in LangChain,"LangChain and pgvector: Up and RunningLangChainis one of the most popular frameworks for building applications and agents with Large Language Models (LLMs). This blog post is an introduction to building LLM applications with the LangChain framework in Python, using PostgreSQL and pgvector as a vector database for OpenAI embeddings data.We'll use the example of creating a chatbot to answer questions about the blog posts from the Timescale blog to illustrate the following concepts:How to prepare your documents for insertion into PostgreSQL and pgvector using LangChain document transformer TextSplitter.How to create embeddings from your data using the OpenAI embeddings modeland insert them into PostgreSQL and pgvector.How to use embeddings retrieved from a vector database to augment LLM generation.This is a great first step for more advanced LangChain projects in Python—for example, creating a chatbot for your company documentation or an application to answer questions from uploaded PDFs.Let's get started!💡Jupyter Notebook and Code:You can find all the code used in this tutorial in a Jupyter Notebook on GitHub in theTimescale Vector Cookbook repo. We recommend cloning the repo and following along by executing the code cells as you read through the tutorial.Setup and ConfigurationSign up for an OpenAI Developer Account and create an API Key. SeeOpenAI's developer platform.Install Python.Install and configure a Python virtual environment. We recommendpyenv.Install the requirements for this notebook using the following command:pip install -r requirements.txtOr, if you already have LangChain installed, runpip install --upgrade langchain.import os
# Run export OPENAI_API_KEY=sk-YOUR_OPENAI_API_KEY...
# Get openAI api key by reading local .env file
from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv())
OPENAI_API_KEY  = os.environ['OPENAI_API_KEY']Next, we need a way for LangChain to interact with PostgreSQL and pgvector. This is achieved by importing the PGVector class from thelangchain.vectorstorespackage as follows.from langchain.vectorstores.pgvector import PGVectorNext, we'll construct our connection string for LangChain to connect to our PostgreSQL database.Because LangChain usesSQLAlchemyto connect to SQL databases like PostgreSQL, we need to create our connection string programmatically, reading each of the components of the string (host, database name, password, port, etc.) from our environment variables.In this example, we'll use a PostgreSQL database with pgvector installed and hosted onTimescale. You can create your own cloud PostgreSQL database in minutesat this linkto follow along.If you're using a Timescale database, you can find all this information in the ""Cheat Sheet"" file you download when creating your new database service. Alternatively, you can also use a local PostgreSQL database if you prefer.# Build the PGVector Connection String from params
# Found in the credential cheat-sheet or ""Connection Info"" in the Timescale console
# In terminal, run: export VAR_NAME=value for each of the values below
host= os.environ['TIMESCALE_HOST']
port= os.environ['TIMESCALE_PORT']
user= os.environ['TIMESCALE_USER']
password= os.environ['TIMESCALE_PASSWORD']
dbname= os.environ['TIMESCALE_DBNAME']

# We use postgresql rather than postgres in the conn string since LangChain uses sqlalchemy under the hood
# You can remove the ?sslmode=require if you have a local PostgreSQL instance running without SSL
CONNECTION_STRING = f""postgresql+psycopg2://{user}:{password}@{host}:{port}/{dbname}?sslmode=require""Ensure you have the pgvector extension installed in your database. You can install it by running. Seesection 2.1 herefor how to install with psycopg2 and python if you prefer.CREATE EXTENSION IF NOT EXISTS vector;Part 1: Use LangChain to split a CSV file into smaller chunks while preserving associated metadataIn this section, we will parse our CSV file into smaller chunks for similarity search and retrieval, with help from LangChains TokenTextSplitter.First, let's take a look at the CSV file we'll be working with:import pandas as pd
import numpy as np
df = pd.read_csv('blog_posts_data.csv')
df.head()titlecontenturl0How to Build a Weather Station With Elixir, Ne...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-to-build-a-...1CloudQuery on Using PostgreSQL for Cloud Asset...This is an installment of our “Community Membe...https://www.timescale.com/blog/cloudquery-on-u...2How a Data Scientist Is Building a Time-Series...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-a-data-scie...3How Conserv Safeguards History: Building an En...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-conserv-saf...4How Messari Uses Data to Open the Cryptoeconom...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-messari-use...As shown above, this is a CSV file ofblog posts about Timescale use cases, in which the developers behind each project explain more about their data infra goals, how they used Timescale to achieve them and share success tips.Ordinarily, we would use the LangChainCSVLoaderto load the contents of a CSV file. But, in this case, we need to pre-process the content column of our CSV to be able to create embeddings for each blog post within the token limits of the OpenAI embeddings API.We also need a way to split the text of the content column of the CSV while retaining the associated metadata with that text (i.e., the blog title and URL).LangChain has several built-in document transformers that make it easy to split, combine, filter, and otherwise manipulate documents.We'll use LangChain'sToken Text Splitterto help us split up the content column of our CSV into chunks of a specified token amount. Alternatively, you can use theRecursive Character Text Splitterif you'd rather split text by the number of characters rather than tokens.We will split the text into chunks of around 512 tokens, with a 20 % or 103 token overlap.import tiktoken
from langchain.text_splitter import TokenTextSplitter
# Split text into chunks of 512 tokens, with 20% token overlap
text_splitter = TokenTextSplitter(chunk_size=512,chunk_overlap=103)Here’s how we’ll split up the chunks:# Helper func: calculate number of tokens
def num_tokens_from_string(string: str, encoding_name = ""cl100k_base"") -> int:
    if not string:
        return 0
    # Returns the number of tokens in a text string
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

#list for smaller chunked text and metadata
new_list = []

# Create a new list by splitting up text into token sizes of around 512 tokens
for i in range(len(df.index)):
    text = df['content'][i]
    token_len = num_tokens_from_string(text)
    if token_len <= 512:
        new_list.append([df['title'][i],
        df['content'][i], 
        df['url'][i]])
    else:
        #split text into chunks using text splitter
        split_text = text_splitter.split_text(text)
        for j in range(len(split_text)):
            new_list.append([df['title'][i],
            split_text[j],
            df['url'][i]])Let's take a look at how the content looks after being split:df_new = pd.DataFrame(new_list, columns=['title', 'content', 'url'])
df_new.head()Part 2: Insert OpenAI embeddings into PostgreSQL and pgvectorNow that we have our original CSV split up into smaller chunks and the associated metadata preserved, we will use the LangChainPandas DataFrame Loaderto load data from our new Pandas data frame and insert it into our PostgreSQL database with pgvector installed.Note that we must specify which column in the data frame contains the text we'll create embeddings for.#load documents from Pandas dataframe for insertion into database
from langchain.document_loaders import DataFrameLoader

# page_content_column is the column name in the dataframe to create embeddings for
loader = DataFrameLoader(df_new, page_content_column = 'content')
docs = loader.load()We'll use the OpenAI embeddings model for our documents, so let's import theOpenAIEmbeddingsmodule from thelangchain.embeddingspackage and create an instance.This instance can be used to generate embeddings for text data using the OpenAI API.from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()💡Learn more about embeddings:For more on OpenAI embeddings and how they're used in Nearest Neighbor Search, seethis explainer.Before we create embeddings for all the data in our data frame, let's briefly overview how creating an embedding works.Here's how we create an embedding for a string:# Create OpenAI embedding using LangChain's OpenAIEmbeddings class
query_string = ""PostgreSQL is my favorite database""
embed = embeddings.embed_query(query_string)
print(len(embed)) # Should be 1536, the dimensionality of OpenAI embeddings
print(embed[:5]) # Should be a list of floatsFor the main event, we'll connect to our PostgreSQL database and store the documents we loaded along with their embeddings.Thanks to LangChain, creating the embeddings and storing the data in our PostgreSQL database is a one-command operation!We pass in the following arguments:documents: The documents we loaded from the Pandas Data Frame.embedding: Our instance of the OpenAI embeddings class, the model we'll use to create the embeddings.collection_name: The name of the table we want our embeddings and metadata to live in.distance_strategy: The distance strategy we want to use to calculate the distance between vectors—in our case, we'll use Cosine distance.connection_string: The connection string to our PostgreSQL database, which we constructed in the setup section.# Create a PGVector instance to house the documents and embeddings
from langchain.vectorstores.pgvector import DistanceStrategy
db = PGVector.from_documents(
    documents= docs,
    embedding = embeddings,
    collection_name= ""blog_posts"",
    distance_strategy = DistanceStrategy.COSINE,
    connection_string=CONNECTION_STRING)Now that our data is in the database, let's perform asimilarity searchto fetch the documents most similar to a query:from langchain.schema import Document

# Query for which we want to find semantically similar documents
query = ""Tell me about how Edeva uses Timescale?""

#Fetch the k=3 most similar documents
docs =  db.similarity_search(query, k=3)The query on our database returns a list of LangChain documents, so let's learn how to interact with those:# Interact with a document returned from the similarity search on pgvector
doc = docs[0]

# Access the document's content
doc_content = doc.page_content
# Access the document's metadata object
doc_metadata = doc.metadata

print(""Content snippet:"" + doc_content[:500])
print(""Document title: "" + doc_metadata['title'])
print(""Document url: "" + doc_metadata['url'])Content snippet:map applications. If you are planning to store time-series data, Timescale is the way to go. It makes it easy to get started because it is “just” SQL, and at the same time, you get the important features needed to work with time-series data. I recommend you have a look, especially at continuous aggregations. Think about the whole lifecycle when you start. Will your use cases allow you to use features like compression, or do you need to think about how to store long-term data outside of TimescaleDBDocument title: How Edeva Uses Continuous Aggregations and IoT to Build Smarter CitiesDocument url:https://www.timescale.com/blog/how-edeva-uses-continuous-aggregations-and-iot-to-build-smarter-cities/Part 3: Question answering with Retrieval Augmented GenerationNext, let's tie everything we've learned together and build a simple example of using LangChain for questions answering using an LLM from OpenAI and the most relevant documents the question from our database.This technique is called Retrieval Augmented Generation (RAG) and works as follows:Create an embedding vector for the user question.Use pgvector to perform a vector similarity search and retrieve the k nearest neighbors to the question embedding from our database of embedding vectors representing the blog content. In our example, we’ll use k=3, finding the three most similar embedding vectors and associated content.Supply the content retrieved from the database as additional context to the model and ask it to perform a completion task to answer the user question.To more easily retrieve documents from our PostgreSQL vector database, we'll use a LangChainretriever.In LangChain, a retriever is an interface that returns documents given an unstructured query. A retriever's main purpose is only to return (or retrieve) documents.We will use avector store-backed retrieverwhich is a retriever that uses a vector store to retrieve documents. It is a lightweight wrapper around the Vector Store class to make it conform to the retriever interface. It uses the search methods implemented by a vector store, like similarity search, to query the texts in the vector store.# Create retriever from database
# We specify the number of results we want to retrieve (k=3)
retriever = db.as_retriever(
    search_kwargs={""k"": 3}
    )Next, we'll import the LLM we want to use to generate a response to our question. In this case, we'll useOpenAI's GPT-3.5 modelwith a 16k token context window so that we won't have any trouble fitting in retrieved documents as context in addition to the user question.from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(temperature = 0.0, model = 'gpt-3.5-turbo-16k')Then, we'll use one of the most useful chains in LangChain, theRetrieval Q+A chain, which is used for question answering over a vector database (vector store or index, as it’s also known).We'll combine it with astuff chain, which takes a list of documents, inserts them all into a prompt (stuffsthem in), and passes that prompt to an LLM.from langchain.chains import RetrievalQA
qa_stuff = RetrievalQA.from_chain_type(
    llm=llm, 
    chain_type=""stuff"", 
    retriever=retriever,
    verbose=True,
)And for the final ingredient, let's formulate a question we want the model to answer with the help of the documents in our database and pass it to our chain to process.query =  ""How does Edeva use continuous aggregates?""

response = qa_stuff.run(query)

from IPython.display import Markdown, display
display(Markdown(response))Edeva uses continuous aggregates in their smart city platform, EdevaLive. They collect large amounts of data from IoT devices, including traffic flow data from their dynamic speed bump called Actibump. Continuous aggregates allow them to roll up multiple resolutions of their sensor account data and people count data, making it available in a more efficient way. This helps them analyze and visualize the data faster, enabling them to provide valuable remote monitoring services and statistics to their customers. They also use continuous aggregates to roll up high-resolution data to lower resolutions, optimizing their data processing.Bonus: Cite Your Sources With LangChain and pgvector for RAGFor even more advanced functionality, you might want your answer to include the sources used to give users peace of mind. Here's how you can do that with the RetrievalQA chain using thereturn_source_documentsargument:# New chain to return context and sources
qa_stuff_with_sources = RetrievalQA.from_chain_type(
    llm=llm, 
    chain_type=""stuff"", 
    retriever=retriever,
    return_source_documents=True,
    verbose=True,
)

query =  ""How does Edeva use continuous aggregates?""

# To run the query, we use a different syntax since we're returning more than just the response text
responses = qa_stuff_with_sources({""query"": query})And finally, let's print out the result with the source document cited:source_documents = responses[""source_documents""]
source_content = [doc.page_content for doc in source_documents]
source_metadata = [doc.metadata for doc in source_documents]

# Construct a single string with the LLM output and the source titles and urls
def construct_result_with_sources():
    result = responses['result']
    result += ""\n\n""
    result += ""Sources used:""
    for i in range(len(source_content)):
    
    result += ""\n\n""
        result += source_metadata[i]['title']
        result += ""\n\n""
        result += source_metadata[i]['url']
    return result

display(Markdown(construct_result_with_sources()))Edeva uses continuous aggregates in their smart city platform, EdevaLive. They collect large amounts of data from IoT devices, including traffic flow data from their dynamic speed bump called Actibump. Continuous aggregates allow them to roll up multiple resolutions of their sensor account data and people count data, making it available in a more efficient way. This helps them analyze and visualize the data faster, enabling them to provide valuable remote monitoring services and statistics to their customers. They also use continuous aggregates to roll up high-resolution data to lower resolutions, optimizing their data processing.Sources used:How Edeva Uses Continuous Aggregations and IoT to Build Smarter Citieshttps://www.timescale.com/blog/how-edeva-uses-continuous-aggregations-and-iot-to-build-smarter-cities/How Density Manages Large Real Estate Portfolios Using TimescaleDBhttps://www.timescale.com/blog/density-measures-large-real-estate-portfolios-using-timescaledb/How Edeva Uses Continuous Aggregations and IoT to Build Smarter Citieshttps://www.timescale.com/blog/how-edeva-uses-continuous-aggregations-and-iot-to-build-smarter-cities/The “cite your sources” functionality is helpful because it can help explain unexpected responses from the model due to irrelevant but highly similar documents being retrieved from the database.Next Steps💡Try it yourself:You can find all the code used in this tutorial in a Jupyter Notebook on GitHub in theTimescale Vector Cookbook repo. Clone the repo and try running it yourself!Check outConversational Retrieval QA Chainfor adding memory and using what you learned in a chatbot conversation setting.Learn how pgvector finds approximate nearest neighbors in this blog post:What Are ivfflat Indexes in pgvector and How Do They Work.UseChainlitto build your own LLM chatbot in Python (Timescale tutorial coming soon!).And if you’re looking for a production PostgreSQL database for your vector workloads,try Timescale. It’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-build-llm-applications-with-pgvector-vector-store-in-langchain/
2023-10-13T13:44:20.000Z,Timescale Vector x LlamaIndex: Making PostgreSQL a Better Vector Database for AI Applications,"Introducing theTimescale Vectorintegration for LlamaIndex. Timescale Vector enables LlamaIndex developers to build better AI applications with PostgreSQL as their vector database: with faster vector similarity search, efficient time-based search filtering, and the operational simplicity of a single, easy-to-use cloud PostgreSQL database for not only vector embeddings but an AI application’s relational and time-series data too.(This post was originally published in LlamaIndex's blog.)PostgreSQL is the world’s most loved database, according to theStack Overflow 2023 Developer Survey. And for a good reason: it’s been battle-hardened by production use for over three decades, it’s robust and reliable, and it has a rich ecosystem of tools, drivers, and connectors.And while pgvector, the open-source extension for vector data on PostgreSQL, is a wonderful extension (and all its features are offered as part of Timescale Vector), it is just one piece of the puzzle in providing a production-grade experience for AI application developers on PostgreSQL.After speaking with numerous developers at nimble startups and established industry giants, we saw the need to enhance pgvector to cater to the performance and operational needs of developers building AI applications.Here’s the TL;DR on how Timescale Vector helps you build better AI applications with LlamaIndex:Faster similarity search on millions of vectors:Thanks to the introduction of a new search index inspired by the DiskANN algorithm,Timescale Vector achieves 3x faster search speed at ~99 % recall than a specialized databaseand outperforms all existing PostgreSQL search indexes by between 39.39 % and 1,590.33 % on a dataset of one million OpenAI embeddings. Plus, enabling product quantization yields a10x index space savings compared to pgvector. Timescale Vector also offers pgvector’s Hierarchical Navigable Small Worlds (HNSW) and Inverted File Flat (IVFFlat) indexing algorithms.Efficient similarity search with time-based filtering:Timescale Vector optimizes time-based vector search queries, leveraging the automatic time-based partitioning and indexing ofTimescale’s hypertablesto efficiently find recent embeddings, constrain vector search by a time range or document age, and store and retrieve large language model (LLM) response and chat history with ease. Time-based semantic search also enables you to use Retrieval Augmented Generation (RAG) with time-based context retrieval to give users more useful LLM responses.Simplified AI infra stack:By combining vector embeddings, relational data, and time-series data in one PostgreSQL database, Timescale Vector eliminates the operational complexity that comes with managing multiple database systems at scale.Simplified metadata handling and multi-attribute filtering:Developers can leverage all PostgreSQL data types to store and filter metadata and JOIN vector search results with relational data for more contextually relevant responses. In future releases, Timescale Vector will further optimize rich multi-attribute filtering, enabling even faster similarity searches when filtering on metadata.On top of these innovations for vector workloads, Timescale Vector provides a robust, production-ready cloud PostgreSQL platform with flexible pricing, enterprise-grade security, and free expert support.In the rest of this post, we’ll dive deeper (with code!) into the unique capabilities Timescale Vector enables for developers wanting to use PostgreSQL as their vector database with LlamaIndex:Faster similarity search with DiskANN, HNSW, and IVFFlat index types.Efficient similarity search when filtering vectors by time.Retrieval Augmented Generation (RAG) with time-based context retrieval.(If you want to jump straight to the code, explorethis tutorial.)🎉LlamaIndex Users Get Three Months of Timescale Vector for Free!We’re giving LlamaIndex users an extended 90-day trial of Timescale Vector. This makes it easy to test and develop your applications on Timescale Vector, as you won’t be charged for any cloud PostgreSQL databases you spin up during your trial period.Try Timescale Vector for free today.Faster Vector Similarity Search in PostgreSQLTimescale Vector speeds up Approximate Nearest Neighbor (ANN) search on large-scale vector datasets, enhancing pgvector with a state-of-the-art ANN index inspired by theDiskANNalgorithm. Timescale Vector also offers pgvector’s HNSW and IVFFlat indexing algorithms, giving developers the flexibility to choose the right index for their use case.Our performance benchmarks using theANN benchmarkssuite show that Timescale Vector achieves between 39.43 % and 1,590.33 % faster search speed at ~99 % recall than all existing PostgreSQL search indexes and 3x faster search speed at ~99 % recall than specialized vector databases on a dataset of one million OpenAI embeddings. You canread more about the performance benchmark methodology, the databases compared,and the results here.Timescale Vector’s new DiskANN-inspired index outperforms all existing PostgreSQL index types when performing approximate nearest neighbor searches at 99 % recall on one million OpenAI embeddingsUsing Timescale Vector’s DiskANN, HNSW, or IVFFLAT indexes in LlamaIndex is incredibly straightforward. Simply create a Timescale Vector vector store and add thedata nodesyou want to query as shown below:from llama_index.vector_stores import TimescaleVectorStore

# Create a timescale vector store with specified params
ts_vector_store = TimescaleVectorStore.from_params(
   service_url=TIMESCALE_SERVICE_URL,
   table_name=""your_table_name"",
   time_partition_interval= timedelta(days=7),
)
ts_vector_store.add(nodes)Then run:# Create a timescale vector index (DiskANN)
ts_vector_store.create_index()This will create a timescale-vector index with the default parameters. We should point out that the term “index” is a bit overloaded. For many VectorStores, an index is the thing that stores your data (in relational databases, this is often called a table), but in the PostgreSQL world, an index is something that speeds up search, and we are using the latter meaning here.We can also specify the exact parameters for index creation in thecreate_indexcommand as follows:# create new timescale vector index (DiskANN) with specified parameters
ts_vector_store.create_index(""tsv"", max_alpha=1.0, num_neighbors=50)Advantages of this Timescale Vector’s new DiskANN-inspired vector search index include the following:Faster vector search at 99 % accuracy in PostgreSQL.Optimized for running on disks, not only in memory use.Quantization optimization compatible with PostgreSQL, reducing the vector size and consequently shrinking the index size (by 10x in some cases!) and expediting searches.Efficient hybrid search or filtering additional dimensions.For more on how Timescale Vector’s new index works,see this blog post.Pgvector is packaged as part of Timescale Vector, so you can also access pgvector’s HNSW and IVFFLAT indexing algorithms in your LlamaIndex applications. The ability to conveniently create ANN search indexes from your LlamaIndex application code makes it easy to create different indexes and compare their performance:# Create an HNSW index
# Note: You don't need to specify m and ef_construction parameters as we set smart defaults.
ts_vector_store.create_index(""hnsw"", m=16, ef_construction=64)

# Create an IVFFLAT index
# Note: You don't need to specify num_lists and num_records parameters as we set smart defaults.
ts_vector_store.create_index(""ivfflat"", num_lists=20, num_records=1000)Add Efficient Time-Based Search Functionality to Your LlamaIndex AI ApplicationTimescale Vector optimizes time-based vector search,leveraging the automatic time-based partitioning and indexing ofTimescale’s hypertablesto efficiently search vectors by time and similarity.Time is often an important metadata component for vector embeddings. Sources of embeddings, like documents, images, and web pages, often have a timestamp associated with them, for example, their creation date, publishing date, or the date they were last updated, to name but a few.We can take advantage of this time metadata in our collections of vector embeddings to enrich the quality and applicability of search results by retrieving vectors that are not just semantically similar but also pertinent to a specific time frame.Here are some examples where time-based retrieval of vectors can improve your LlamaIndex applications:Finding recent embeddings:Finding the most recent embeddings that are semantically similar to a query vector—for example, finding the most recent news, documents, or social media posts related to elections.Search within a time range:Constraining similarity search to only vectors within a relevant time range. For example, asking time-based questions about a knowledge base (“What new features were added between January and March 2023?”).Chat history:Storing and retrieving LLM response history. For example, chatbot chat history.Let’s take a look at an example of performing time-based searches on agit log dataset. In a git log, each entry has a timestamp, an author, and some information about the commit.To illustrate how to use TimescaleVector's time-based vector search functionality, we'll ask questions about the git log history for TimescaleDB. Each git commit entry has a timestamp associated with it, as well as a message and other metadata (e.g., author).We'll illustrate how to create nodes with a time-based UUID and how to run similarity searches with time range filters using the Timescale Vector vector store.Create nodes from each commit in the git logFirst, we load the git log entries from thedemo CSV fileusing Pandas:import pandas as pd
from pathlib import Path


# Read the CSV file into a DataFrame
file_path = Path(""../data/csv/commit_history.csv"")
df = pd.read_csv(file_path)Next, we’ll create nodes of typeTextNodefor each commit in our git log dataset, extracting the relevant information and assigning it to the node’s text and metadata, respectively.from llama_index.schema import TextNode, NodeRelationship, RelatedNodeInfo
# Create a Node object from a single row of data
def create_node(row):
   record = row.to_dict()
   record_name = split_name(record[""author""])
   record_content = str(record[""date""]) + "" "" + record_name + "" "" + str(record[""change summary""]) + "" "" + str(record[""change details""])
   node = TextNode(
       id_=create_uuid(record[""date""]),
       text= record_content,
       metadata={
           'commit': record[""commit""],
           'author': record_name,
           'date': create_date(record[""date""]),
       }
   )
   return node

nodes = [create_node(row) for _, row in df.iterrows()]Note:The code above references two helper functions to get things in the right format (split_name()andcreate_date()), which we’ve omitted for brevity. The full code is included in the tutorial linked in the Resources section at the end of this post.Create UUIDs for each node based on the date of each git commitWe will take a closer look at a helper function we use to create each node’sid_. For time-based search in LlamaIndex, Timescale Vector uses the ‘datetime’ portion of a UUID v1 to place vectors in the correct time partition.Timescale Vector’s Python client libraryprovides a simple-to-use function nameduuid_from_timeto create a UUID v1 from a Python DateTime object, which we’ll then use as ouridsfor the TextNodes.from timescale_vector import client
# Function to take in a date string in the past and return a uuid v1
def create_uuid(date_string: str):
   if date_string is None:
       return None
   time_format = '%a %b %d %H:%M:%S %Y %z'
   datetime_obj = datetime.strptime(date_string, time_format)
   uuid = client.uuid_from_time(datetime_obj)
   return str(uuid)Since we are dealing with timestamps in the past, we take advantage of theuuid_from_timefunction to help generate the correct UUIDs for each node. If you want the current date and time associated with your Nodes (or Documents) for time-based search, you can skip this step. A UUID associated with the current date and time will be automatically generated as the nodes are added to the table in Timescale Vector by default.Let’s take a look at the contents of a node:print(nodes[0].get_content(metadata_mode=""all""))commit: 44e41c12ab25e36c202f58e068ced262eadc8d16
author: Lakshmi Narayanan Sreethar
date: 2023-09-5 21:03:21+0850

Tue Sep 5 21:03:21 2023 +0530 Lakshmi Narayanan Sreethar Fix segfault in set_integer_now_func When an invalid function oid is passed to set_integer_now_func, it finds out that the function oid is invalid but before throwing the error, it calls ReleaseSysCache on an invalid tuple causing a segfault. Fixed that by removing the invalid call to ReleaseSysCache.  Fixes #6037Create vector embeddings for the text of each nodeNext, we'll create vector embeddings of the content of each node so that we can perform similarity searches on the text associated with each node. We'll use theOpenAIEmbeddingmodel to create the embeddings.# Create embeddings for nodes
from llama_index.embeddings import OpenAIEmbedding
embedding_model = OpenAIEmbedding()

for node in nodes:
   node_embedding = embedding_model.get_text_embedding(
       node.get_content(metadata_mode=""all"")
   )
   node.embedding = node_embeddingLoad nodes into Timescale Vector vector storeNext, we'll create aTimescaleVectorStoreinstance and add the nodes we created to it.# Create a timescale vector store and add the newly created nodes to it
ts_vector_store = TimescaleVectorStore.from_params(
   service_url=TIMESCALE_SERVICE_URL,
   table_name=""li_commit_history"",
   time_partition_interval= timedelta(days=7),
)
ts_vector_store.add(nodes)To take advantage of Timescale Vector’s efficient time-based search, we need to specify thetime_partition_intervalargument when instantiating a Timescale Vector vector store. This argument represents the length of each interval for partitioning the data by time. Each partition will consist of data that falls within the specified length of time.In the example above, we use seven days for simplicity, but you can pick whatever value makes sense for the queries used by your application—for example, if you query recent vectors frequently, you might want to use a smaller time delta like one day, or if you query vectors over a decade-long time period, then you might want to use a larger time delta like six months or one year.As a rule of thumb, common queries should touch only a couple of partitions, and at the same time, your full dataset should fit within 1,000 partitions, but don’t stress too much—the system is not very sensitive to this value.Similarity search with time filtersNow that we’ve loaded our nodes that contain vector embedding data and metadata into a Timescale Vector vector store and enabled automatic time-based partitioning on the table our vectors and metadata are stored in, we can query our vector store with time-based filters as follows:# Query the vector database
vector_store_query = VectorStoreQuery(query_embedding = query_embedding, similarity_top_k=5)

# Time filter variables for query
start_dt = datetime(2023, 8, 1, 22, 10, 35) # Start date = 1 August 2023, 22:10:35
end_dt = datetime(2023, 8, 30, 22, 10, 35) # End date = 30 August 2023, 22:10:35

# return most similar vectors to query between start date and end date date range
# returns a VectorStoreQueryResult object
query_result = ts_vector_store.query(vector_store_query, start_date = start_dt, end_date = end_dt)Let’s take a look at the date and contents of the nodes returned by our query:# for each node in the query result, print the node metadata date
for node in query_result.nodes:
   print(""-"" * 80)
   print(node.metadata[""date""])
   print(node.get_content(metadata_mode=""all""))--------------------------------------------------------------------------------
2023-08-3 14:30:23+0500
commit:  7aeed663b9c0f337b530fd6cad47704a51a9b2ec
author: Dmitry Simonenko
date: 2023-08-3 14:30:23+0500

Thu Aug 3 14:30:23 2023 +0300 Dmitry Simonenko Feature flags for TimescaleDB features This PR adds..
--------------------------------------------------------------------------------
2023-08-29 18:13:24+0320
commit:  e4facda540286b0affba47ccc63959fefe2a7b26
author: Sven Klemm
date: 2023-08-29 18:13:24+0320

Tue Aug 29 18:13:24 2023 +0200 Sven Klemm Add compatibility layer for _timescaledb_internal functions With timescaledb 2.12 all the functions present in _timescaledb_internal were…
--------------------------------------------------------------------------------
2023-08-22 12:01:19+0320
commit:  cf04496e4b4237440274eb25e4e02472fc4e06fc
author: Sven Klemm
date: 2023-08-22 12:01:19+0320

Tue Aug 22 12:01:19 2023 +0200 Sven Klemm Move utility functions to _timescaledb_functions schema To increase schema security we do not want to mix…
--------------------------------------------------------------------------------
2023-08-29 10:49:47+0320
commit:  a9751ccd5eb030026d7b975d22753f5964972389
author: Sven Klemm
date: 2023-08-29 10:49:47+0320

Tue Aug 29 10:49:47 2023 +0200 Sven Klemm Move partitioning functions to _timescaledb_functions schema To increase schema security…
--------------------------------------------------------------------------------
2023-08-9 15:26:03+0500
commit:  44eab9cf9bef34274c88efd37a750eaa74cd8044
author: Konstantina Skovola
date: 2023-08-9 15:26:03+0500

Wed Aug 9 15:26:03 2023 +0300 Konstantina Skovola Release 2.11.2 This release contains bug fixes since the 2.11.1 release…Success! Notice how only vectors with timestamps within the specified start and end date ranges of 1 August 2023, and 30 August 2023, are included in the results.Here’s some intuition for why Timescale Vector’s time-based partitioning speeds up ANN queries with time-based filters.Timescale Vector partitions the data by time and creates ANN indexes on each partition individually. Then, during search, we perform a three-step process:Step 1: filter our partitions that don’t match the time predicate.Step 2: perform the similarity search on all matching partitions.Step 3: combine all the results from each partition in step 2, rerank, and filter out results by time.Timescale Vector leveragesTimescaleDB’s hypertables, which automatically partition vectors and associated metadata by a timestamp. This enables efficient querying on vectors by both similarity to a query vector and time, as partitions not in the time window of the query are ignored, making the search a lot more efficient by filtering out whole swaths of data in one go.When performing a vector similarity search onTimescaleVectorStoreembedding, rather than specifying the start and end dates for our search, we can also specify a time filter with a provided start date and time delta later:# return most similar vectors to query from start date and a time delta later
query_result = ts_vector_store.query(vector_store_query, start_date = start_dt, time_delta = td)And we can specify a time filter within a providedend_dateandtime_deltaearlier. This syntax is very useful for filtering your search results to contain vectors before a certain date cutoff.# return most similar vectors to query from end date and a time delta earlier
query_result = ts_vector_store.query(vector_store_query, end_date = end_dt, time_delta = td)Powering Retrieval Augmented Generation With Time-Based Context Retrieval in LlamaIndex Applications With Timescale VectorLet’s put everything together and look at how to use the TimescaleVectorStore to power RAG on the git log dataset we examined above.To do this, we can use the TimescaleVectorStore as aQueryEngine. When creating the query engine, we use TimescaleVector's time filters to constrain the search to a relevant time range by passing our time filter parameters asvector_strore_kwargs.from llama_index import VectorStoreIndex
from llama_index.storage import StorageContext

index = VectorStoreIndex.from_vector_store(ts_vector_store)
query_engine = index.as_query_engine(vector_store_kwargs = ({""start_date"": start_dt, ""end_date"":end_dt}))

query_str = ""What's new with TimescaleDB functions? When were these changes made and by whom?""
response = query_engine.query(query_str)
print(str(response))We asked the LLM a question about our git log, namely, “What’s new with TimescaleDB functions? When were these changes made and by whom?”Here’s the response we get, which synthesizes the nodes returned from semantic search with time-based filtering on the Timescale VectorStore:TimescaleDB functions have undergone changes recently. These changes include the addition of several GUCs (Global User Configuration) that allow for enabling or disabling major TimescaleDB features. Additionally, a compatibility layer has been added for the ""_timescaledb_internal"" functions, which were moved into the ""_timescaledb_functions"" schema to enhance schema security. These changes were made by Dmitry Simonenko and Sven Klemm. The specific dates of these changes are August 3, 2023, and August 29, 2023, respectively.This is a simple example of a powerful concept—using time-based context retrieval in your RAG applications can help provide more relevant answers to your users. This time-based context retrieval can be helpful to any dataset with a natural language and time component. Timescale Vector uniquely enables this thanks to its efficient time-based similarity search capabilities, and taking advantage of it in your LlamaIndex application is easy thanks to the Timescale Vector integration.Resources and next stepsNow that you’ve learned how Timescale Vector can help you power better AI applications with PostgreSQL, it’s your turn to dive in. Take the next step in your learning journey by following one of the tutorials or reading one of the blog posts in the resource set below:Up and Running Tutorial:learn how to use Timescale Vector in LlamaIndex using a real-world dataset. You’ll learn how to use Timescale Vector as a Vectorstore, Retriever, and QueryEngine and perform time-based similarity search on vectors.Timescale Vector explainer: learn more about the internals of Timescale Vector.Timescale Vector website:learn more about Timescale Vector and Timescale’s AI Launch Week.🎉 And a reminder:LlamaIndex users get Timescale Vector free for 90 days.We’re giving LlamaIndex users an extended 90-day trial of Timescale Vector. This makes it easy to test and develop your applications on Timescale Vector, as you won’t be charged for any cloud PostgreSQL databases you spin up during your trial period.Try Timescale Vector for free today.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescale-vector-x-llamaindex-making-postgresql-a-better-vector-database-for-ai-applications/
2023-07-26T13:00:22.000Z,Supercharge Your AI Agents With Postgres: An Experiment With OpenAI's GPT-4,"Hello developers, AI enthusiasts, and everyone eager to push the boundaries of what's possible with technology! Today, we're exploringAI agentsas intermediaries in a fascinating intersection of fields: Artificial Intelligence and databases.The Dawn of AI AgentsAI agents are at the heart of the tech industry's ongoing revolution. As programs capable of autonomous actions in their environment, AI agents analyze, make decisions, and execute actions that drive a myriad of applications. From autonomous vehicles and voice assistants to recommendation systems and customer service bots, AI agents are changing the way we interact with technology.But what if we could take it a step further? What if we could use AI to simplify how we interact with databases? Could AI agents act as intermediaries, interpreting human language and converting it into structured database queries?A Ruby Experiment With GPT-4That's exactly what we tried to achieve in a recent experiment. Leveraging OpenAI's GPT-4, a powerful language model, we conducted an experiment to see how we could use AI to interact with our databases using everyday language.The experiment was built using Ruby, and you can find thedetailed explanation and code here. The results were fascinating, revealing the potential power of using AI as a “middle-man” (Middle-tech? Middle-bot?) between humans and databases.Check out the videos throughout this blog postto see it in action:Why Store Data for AI Agents?Data storage is crucial for the successful application of AI, particularly for training and fine-tuning models. By storing interactions, results, and other relevant data, we can improve the performance and accuracy of our AI agents over time.But data storage is not just about improving our AI; it's also about cost-effectiveness. With the OpenAI API, you pay per token, which can add up when dealing with large amounts of data. By using PostgreSQL as long-term memory for your AI agent, you can reduce the number of tokens you send to the OpenAI API, saving computational resources and money.PostgreSQL: Flexible and RobustPostgreSQL is a powerful, open-source relational database system. With a reputation for reliability, robustness, and performance, it's a fantastic choice for your AI's long-term memory. PostgreSQL also offers flexibility and scalability, making it suitable for projects of all sizes.Whether you're conducting experiments or deploying production-ready applications, PostgreSQL's flexibility and robust nature make it an excellent companion for your AI.Needless to say, we’re huge PostgreSQL enthusiasts here at Timescale—so much so that we built Timescale on PostgreSQL. Timescale works just like PostgreSQL under the hood, offering the same 100 percent SQL support (not SQL-like) and a rich ecosystem of connectors and tools but supercharging PostgreSQL for analytics, events, and time series (and time-series-like workloads).With additional features likecompressionandautomatically updated incremental materialized views—we call them continuous aggregates—Timescale allows you to scale PostgreSQL further for optimal performance while enjoying the best developer experience and cost-effectiveness.But why all this talk about Timescale? As the conversation between human and machine is happening on point in time, I realize I’m dealing with time-series data. Cue in TimescaleDB for the rescue!Join the Timescale CommunityWe're just scratching the surface of what's possible when combining AI with databases like PostgreSQL, and we'd love for you to join us on this journey.Got a cool idea? A question? Or just want to share your thoughts on this topic? Join the Timescale Community onSlackand head over to the#ai-llm-discussionchannel. Let's push the boundaries together and shape the future of AI!Check this page to learn how topower agents, chatbots, and other large language models AI applications with PostgreSQL. To see what my fellow Timescalers Avthar, Mat, and Sam are already building, read their post onPostgreSQL as a Vector Database: Create, Store, and Query OpenAI Embeddings With pgvector.Remember, technology grows exponentially when great minds come together. See you there!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/supercharge-your-ai-agent-with-postgresql-an-experiment-with-openais-gpt-4/
2023-07-05T14:48:04.000Z,Jupyter Notebook Tutorial: Setting Up Python & Jupyter Notebooks on macOS for OpenAI Exploration,"Are you ready to delve into the world of AI and ChatGPT? Jumping right in is the best way to get started. As the most commonly used programming language in this field,Pythonpairs perfectly with Jupyter Notebooks, a powerful tool for running Python scripts.In this Jupyter Notebook tutorial, we'll guide you through setting up these tools on your macOS system. We will be utilizing the following components:Pyenv: to manage multiple Python installations on your machine, allowing different Python versions for various projects.Virtualenv: to create isolated environments for your Python projects, ensuring independent installations of packages and dependencies.Jupyter Notebook:an interactive computing environment that enables you to create and share documents containing live code, visualizations, explanatory text, and more.Let's start this Jupyter Notebook tutorial by installing the necessary dependencies:brew update
brew install pyenv
brew install pyenv-virtualenv
brew install npm  # JupyterLab requires npmNext, ensure that your shell is correctly utilizing pyenv (assuming you're using Z shell—or Zsh; for other shells, see thepyenv GitHub repo):echo 'export PYENV_ROOT=""$HOME/.pyenv""' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH=""$PYENV_ROOT/bin:$PATH""' >> ~/.zshrc
echo 'eval ""$(pyenv init -)""' >> ~/.zshrc
echo 'if which pyenv-virtualenv-init > /dev/null; then eval ""$(pyenv virtualenv-init -)""; fi' >> ~/.zshrcAfterward, open a new shell to apply the updates and install the latest Python 3:pyenv install 3
pyenv global 3Now, let's create a virtual environment for your project:pyenv virtualenv 3 jupyter_env
pyenv activate jupyter_envAt any time, you can view your virtual environments by running:pyenv virtualenvsInstall Jupyter Notebook in your new environment:pip install jupyterlab
jupyter lab buildCreate a directory for your project:mkdir my_jupyter_project
cd my_jupyter_projectTo install some dependencies, execute the following command:pip install openai python-dotenvLaunch Jupyter Lab:Jupyter-labYou're now ready to explore:Ourtutorialon OpenAI embeddings.TheOpenAI cookbooks.📺Editor's Note:Head to our YouTube channel to learn how to connect your Timescale database from Jupyter Notebooks.Extra, Extra! Storing OpenAI TokensWe recommend storing your OpenAI token in a .env file. To create it, run the following command:echo ""OPENAI_API_KEY=your-api-key"" > .envTo load the token in your notebooks, include the following code:from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv()) 
openai.api_key = os.environ['OPENAI_API_KEY']Now you're all set to make the most of OpenAI's capabilities within your notebooks. We hope you find this Jupyter Notebook tutorial using Python on macOS helpful. Happy exploring!➡️Next steps:Now that you've set up Python and Jupyter notebook, take the next step by learning how tocreate, store and query OpenAI embeddings using PostgreSQL and pgvector.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/jupyter-notebook-tutorial-setup-python-and-jupyter-notebooks-macos/
2023-06-21T18:22:10.000Z,"PostgreSQL as a Vector Database: Create, Store, and Query OpenAI Embeddings With pgvector","Looking for a “Hello world” tutorial for pgvector and OpenAI embeddings that gives you the basics of using PostgreSQL as a vector database? You’ve found it!Vector databases enable efficient storage and search of vector data and are essential to developing and maintaining AI applications using Large Language Models (LLMs).With a little help from thepgvector extension, you can leverage PostgreSQL, the flexible and robust SQL database, as a vector database to store and queryOpenAI embeddings. Used to measure the similarity of text strings, OpenAI embeddings are a type of data representation (in the shape of vectors, i.e., lists of numbers) for OpenAI’s models. Much more on OpenAI embeddings, pgvector and vector databases later in this post.We’ll use the example of creating a chatbot to answer questions about Timescale use cases, referencing content from theTimescale Developer Q&A blog posts, to illustrate the key concepts for creating, storing, and querying OpenAI embeddings with PostgreSQL and pgvector.We divided the post into three parts:Part 1: How to create embeddings from content using theOpenAI API.Part 2: How to use PostgreSQL as a vector database and store OpenAI embedding vectors using pgvector.Part 3: How to use embeddings retrieved from a vector database to augment LLM generation.One could think of this tutorial as a first step to building a chatbot that can reference a company knowledge base or developer docs.✨Jupyter Notebook and Code:You can find all the code used in this tutorial in a Jupyter Notebook, as well as sample content and embeddings on the Timescale GitHub:timescale/vector-cookbook. We recommend cloning the repo and following along by executing the code cells as you read through the tutorial.Why Create and Store OpenAI Embeddings for Your Documents?We will explain how to useRetrieval Augmented Generation (RAG)to create a chatbot that combines your data with the power of ChatGPT using OpenAI and pgvector. RAG addresses the problem that a foundational model (e.g., GPT-3 or GPT-4) may be missing some information needed to give a good answer because that information was not in the dataset used to train the model (for example, the information is stored in private documents or only became available recently).RAG’s solution is dead simple: provide additional context to the foundational model in the prompt. For example, if someone asks a baking chatbot, “What is a cronut?” and the foundational model has never heard of cronuts, you can transform the prompt into context: “A cronut resembles a doughnut and is made from croissant-like dough filled with flavored cream and fried in grapeseed oil. What is a cronut?”The foundational model can then use its knowledge of donuts and croissants to wax eloquently about cronuts. This technique is insanely powerful—it allows you to “teach” foundational models about things only you know about and use that to create a ChatGPT++ experience for your users!But what context do you provide to the model? If you have a library of information, how do you know what’s relevant to a given question? Cue in embeddings. As mentioned above,OpenAI embeddingsare a mathematical representation of the semantic meaning of a piece of text that allows forsimilarity search.This means that if you get a user question and calculate its embedding, you can use similarity search against data embeddings in your library to find the most relevant information. But that requires having an embedding representation of your library.This post is a guide to creating, storing, and querying OpenAI vector embeddings usingpgvector, the extension that turns PostgreSQL into a vector database.What is pgvector?Pgvectoris an open-source extension for PostgreSQL that enables storing and searching over machine learning-generated embeddings. It provides different capabilities that let users identify both exact and approximate nearest neighbors. It is designed to work seamlessly with other PostgreSQL features, including indexing and querying.Let's get started!Before You Begin: Pre-Requisites and ConfigurationInstall Python.Install and configure a Python virtual environment. We recommendPyenv.Install the requirements for this notebook using the following command:pip install -r requirements.txtImport all the packages we will be using:import openai
import os
import pandas as pd
import numpy as np
import json
import tiktoken
import psycopg2
import ast
import pgvector
import math
from psycopg2.extras import execute_values
from pgvector.psycopg2 import register_vectorYou’ll need tosign up for an OpenAI Developer Accountand create an OpenAI API Key – we recommend getting a paid account to avoid rate limiting and settting a spending cap so that you avoid any surprises with bills.Once you have an OpenAI API key, it’s abest practiceto store it as an environment variable and then have your Python program read it.#First, run export OPENAI_API_KEY=sk-YOUR_OPENAI_API_KEY...


# Get openAI api key by reading local .env file
from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv()) 
openai.api_key  = os.environ['OPENAI_API_KEY']Part 1: Create Embeddings for Your Data Using the OpenAI APIEmbeddingsmeasure how related text strings are. First, we'll create embeddings using the OpenAI API on some text we want the LLM to answer questions on.In this example, we'll use content from the Timescale blog, specifically from theDeveloper Q&A section, which features posts by Timescale users talking about their real-world use cases.You can replace this blog data with any text you want to embed, such as your own company blog, developer documentation, internal knowledge base, or any other information you’d like to have a “ChatGPT-like” experience over.# Load your CSV file into a pandas DataFrame
df = pd.read_csv('blog_posts_data.csv')
df.head()The output looks like this:TitleContentURL0How to Build a Weather Station With Elixir, Ne...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-to-build-a-...1CloudQuery on Using PostgreSQL for Cloud Asset...This is an installment of our “Community Membe...https://www.timescale.com/blog/cloudquery-on-u...2How a Data Scientist Is Building a Time-Series...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-a-data-scie...3How Conserv Safeguards History: Building an En...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-conserv-saf...4How Messari Uses Data to Open the Cryptoeconom...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-messari-use...1.1 Calculate the cost of embedding dataIt's usually a good idea to calculate how much creating embeddings for your selected content will cost. We provide a number of helper functions to calculate a cost estimate before creating the embeddings to help us avoid surprises.For OpenAI, you are charged on a per-token basis for embeddings created. The total cost will be less than $0.01 for the blog posts we want to embed, thanks to OpenAI’s recent announcement of a75 % cost reductionin their most popular embedding model,text-embedding-ada-002.What is a token?Tokens are common sequences of characters found in text. Roughly speaking, a token is three-quarters (¾) of a word. Large language models, like GPT-3 and GPT-4 made by OpenAI, are trained to understand the statistical relationships between tokens and predict the next token in a sequence. Learn more about tokens withOpenAI’s Tokenizer tool.# Helper functions to help us create the embeddings

# Helper func: calculate number of tokens
def num_tokens_from_string(string: str, encoding_name = ""cl100k_base"") -> int:
    if not string:
        return 0
    # Returns the number of tokens in a text string
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

# Helper function: calculate length of essay
def get_essay_length(essay):
    word_list = essay.split()
    num_words = len(word_list)
    return num_words

# Helper function: calculate cost of embedding num_tokens
# Assumes we're using the text-embedding-ada-002 model
# See https://openai.com/pricing
def get_embedding_cost(num_tokens):
    return num_tokens/1000*0.0001

# Helper function: calculate total cost of embedding all content in the dataframe
def get_total_embeddings_cost():
    total_tokens = 0
    for i in range(len(df.index)):
        text = df['content'][i]
        token_len = num_tokens_from_string(text)
        total_tokens = total_tokens + token_len
    total_cost = get_embedding_cost(total_tokens)
    return total_cost# quick check on total token amount for price estimation
total_cost = get_total_embeddings_cost()
print(""estimated price to embed this content = $"" + str(total_cost))1.2 Create smaller chunks of contentThe OpenAI API has alimitto the maximum number of tokens it can create an embedding for in a single request: 8,191 to be specific.To get around this limit, we'll break up our text into smaller chunks. Generally, it's a best practice to “chunk” the documents you want to create embeddings into groups of a fixed token size.The precise number of tokens to include in a chunk depends on your use case and your model’s context window—the number of input tokens it can handle in a prompt.For our purposes, we'll aim for chunks of around 512 tokens each. Chunking text up is a complex topic worthy of its own blog post. We’ll illustrate a simple method we found to work well below. If you want to read about other approaches, we recommendthis blog postandthis sectionof the LangChain docs.Note:If you prefer to skip this step, you can use the provided file:blog_data_and_embeddings.csv, which contains the data and embeddings that you'll generate in this step.The code below creates a new list of our blog content while retaining the metadata associated with the text, such as the blog title and URL that the text is associated with.# Create new list with small content chunks to not hit max token limits
# Note: the maximum number of tokens for a single request is 8191
# https://openai.com/docs/api-reference/requests

# list for chunked content and embeddings
new_list = []
# Split up the text into token sizes of around 512 tokens
for i in range(len(df.index)):
    text = df['content'][i]
    token_len = num_tokens_from_string(text)
    if token_len <= 512:
        new_list.append([df['title'][i], df['content'][i], df['url'][i], token_len])
    else:
        # add content to the new list in chunks
        start = 0
        ideal_token_size = 512
        # 1 token ~ 3/4 of a word
        ideal_size = int(ideal_token_size // (4/3))
        end = ideal_size
        #split text by spaces into words
        words = text.split()

        #remove empty spaces
        words = [x for x in words if x != ' ']

        total_words = len(words)
        
        #calculate iterations
        chunks = total_words // ideal_size
        if total_words % ideal_size != 0:
            chunks += 1
        
        new_content = []
        for j in range(chunks):
            if end > total_words:
                end = total_words
            new_content = words[start:end]
            new_content_string = ' '.join(new_content)
            new_content_token_len = num_tokens_from_string(new_content_string)
            if new_content_token_len > 0:
                new_list.append([df['title'][i], new_content_string, df['url'][i], new_content_token_len])
            start += ideal_size
            end += ideal_sizeNow that our text is chunked better, we can create embeddings for each chunk of text using the OpenAI API.We’ll use this helper function to create embeddings for a piece of text:# Helper function: get embeddings for a text
def get_embeddings(text):
   response = openai.Embedding.create(
       model=""text-embedding-ada-002"",
       input = text.replace(""\n"","" "")
   )
   embedding = response['data'][0]['embedding']
   return embeddingAnd then create embeddings for each chunk of content:# Create embeddings for each piece of content
for i in range(len(new_list)):
   text = new_list[i][1]
   embedding = get_embeddings(text)
   new_list[i].append(embedding)

# Create a new dataframe from the list
df_new = pd.DataFrame(new_list, columns=['title', 'content', 'url', 'tokens', 'embeddings'])
df_new.head()The new data frame should look like this:TitleContentURLTokensEmbeddings0How to Build a Weather Station With Elixir, Ne...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-to-build-a-...501[0.021440856158733368, 0.02200360782444477, -0...1How to Build a Weather Station With Elixir, Ne...capture weather and environmental data. In all...https://www.timescale.com/blog/how-to-build-a-...512[0.016165969893336296, 0.011341351084411144, 0...2How to Build a Weather Station With Elixir, Ne...command in their database migration:SELECT cre...https://www.timescale.com/blog/how-to-build-a-...374[0.022517921403050423, -0.0019158280920237303,...3CloudQuery on Using PostgreSQL for Cloud Asset...This is an installment of our “Community Membe...https://www.timescale.com/blog/cloudquery-on-u...519[0.009028822183609009, -0.005185891408473253, ...4CloudQuery on Using PostgreSQL for Cloud Asset...Architecture with CloudQuery SDK- Writing plug...https://www.timescale.com/blog/cloudquery-on-u...511[0.02050386555492878, 0.010169642977416515, 0....As an optional but recommended step, you can save the original blog content along with associated embeddings in a CSV file for reference later on so that you don't have to recreate embeddings if you want to reference it in another project.# Save the dataframe with embeddings as a CSV file
df_new.to_csv('blog_data_and_embeddings.csv', index=False)Part 2: Store OpenAI Embeddings in a Vector Database Using pgvectorNow that we have created embedding vectors for our blog content, the next step is to store the embedding vectors in a vector database to help us perform a fast search over many vectors.What is a vector database?A vector database is a database that can handle vector data.Vector databases are useful for:Semantic search:Vector databases facilitate semantic search, which considers the context or meaning of search terms rather than just exact matches. They are useful for recommendation systems, content discovery, and question-answering systems.Efficient similarity search:Vector databases are designed for efficient high-dimensional nearest neighbor search, a task where traditional relational databases struggle.Machine learning:Vector databases store and search embeddings created by machine-learning models. This feature aids in finding items semantically similar to a given item.Multimedia data handling:Vector databases also excel in working with multimedia data (images, audio, video) by converting them into high-dimensional vectors for efficient similarity search.NLP and data combination:In Natural Language Processing (NLP), vector databases store high-dimensional vectors representing words, sentences, or documents. They also allow a combination of traditional SQL queries with similarity searches, accommodating both structured and unstructured data.We’ll use PostgreSQL with thepgvector extensioninstalled as our vector database. Pgvector extends PostgreSQL to handle vector data types and vector similarity search, likenearest neighbor search, which we’ll use to find thekmost related embeddings in our database for a given user prompt.Why use pgvector as a vector database?Here are five reasons why PostgreSQL is a good choice for storing and handling vector data:Integrated solution:By using PostgreSQL as a vector database, you keep your data in one place. This can simplify your architecture by reducing the need for multiple databases or additional services.Enterprise-level robustness and operations:With a 30-year pedigree, PostgreSQL provides world-class data integrity, operations, and robustness. This includes backups, streaming replication, role-based and row-level security, and ACID compliance.Full-featured SQL:PostgreSQL supports a rich set of SQL features, including joins, subqueries, window functions, and more. This allows for powerful and complex queries that can include both traditional relational data and vector data. It also integrates with a plethora of existing data science and data analysis tools.Scalability and performance:PostgreSQL is known for its robustness and ability to handle large datasets. Using it as a vector database allows you to leverage these characteristics for vector data as well.Open source:PostgreSQL is open source, which means it's free to download and use, and you can modify it to suit your needs. It also means that it benefits from the collective input of developers all over the world, which often results in high-quality, secure, and up-to-date software. PostgreSQL has a large and active community, so help is readily available. There are many resources, such as documentation, tutorials, forums, and more, to help you troubleshoot and optimize your PostgreSQL database.2.1 Create a PostgreSQL database and install pgvectorFirst, we’ll create a PostgreSQL database. You cancreate a cloud PostgreSQL databasein minutes for free onTimescaleor use a local PostgreSQL database for this step.Once you’ve created your PostgreSQL database, export your connection string as an environment variable, and just like the OpenAI API key, we’ll read it into our Python program from the environment file:# Timescale database connection string
# Found under ""Service URL"" of the credential cheat-sheet or ""Connection Info"" in the Timescale console
# In terminal, run: export TIMESCALE_CONNECTION_STRING=postgres://<fill in here>

connection_string  = os.environ['TIMESCALE_CONNECTION_STRING']We then connect to our database using the popularpsycopg2python library  and install the pgvector extension as follows:# Connect to PostgreSQL database in Timescale using connection string
conn = psycopg2.connect(connection_string)
cur = conn.cursor()

#install pgvector
cur.execute(""CREATE EXTENSION IF NOT EXISTS vector"");
conn.commit()2.2 Connect to and configure your vector databaseOnce we’ve installed pgvector, we use theregister_vector()command to register the vector type with our connection:# Register the vector type with psycopg2
register_vector(conn)Once we’ve connected to the database, let’s create a table that we’ll use to store embeddings along with metadata. Our table will look as follows:idtitleurlcontenttokensembeddingIdrepresents the unique ID of each vector embedding in the table.titleis the blog title from which the content associated with the embedding is taken.urlis the blog URL from which the content associated with the embedding is taken.contentis the actual blog content associated with the embedding.tokensis the number of tokens the embedding represents.embeddingis the vector representation of the content.One advantage of using PostgreSQL as a vector database is that you can easily store metadata and embedding vectors in the same database, which is helpful for supplying the user-relevant information related to the response they receive, like links to read more or specific parts of a blog post that are relevant to them.# Create table to store embeddings and metadata
table_create_command = """"""
CREATE TABLE embeddings (
            id bigserial primary key, 
            title text,
            url text,
            content text,
            tokens integer,
            embedding vector(1536)
            );
            """"""

cur.execute(table_create_command)
cur.close()
conn.commit()2.3 Ingest and store vector data into PostgreSQL using pgvectorNow that we’ve created the database and created the table to house the embeddings and metadata, the final step is to insert the embedding vectors into the database.For this step, it’s a best practice to batch insert the embeddings rather than insert them one by one.#Batch insert embeddings and metadata from dataframe into PostgreSQL database
register_vector(conn)
cur = conn.cursor()
# Prepare the list of tuples to insert
data_list = [(row['title'], row['url'], row['content'], int(row['tokens']), np.array(row['embeddings'])) for index, row in df_new.iterrows()]
# Use execute_values to perform batch insertion
execute_values(cur, ""INSERT INTO embeddings (title, url, content, tokens, embedding) VALUES %s"", data_list)
# Commit after we insert all embeddings
conn.commit()Let’s sanity check by running some simple queries against our newly inserted data:cur.execute(""SELECT COUNT(*) as cnt FROM embeddings;"")
num_records = cur.fetchone()[0]
print(""Number of vector records in table: "", num_records,""\n"")
# Correct output should be 129# print the first record in the table, for sanity-checking
cur.execute(""SELECT * FROM embeddings LIMIT 1;"")
records = cur.fetchall()
print(""First record in table: "", records)2.4 Index your data for faster retrievalIn this example, we only have 129 embedding vectors, so searching through all of them is blazingly fast. But for larger datasets, you need to create indexes to speed up searching for similar embeddings, so we include the code to build the index for illustrative purposes.Pgvector supports the ivfflat index type to provide for speed up of approximate nearest neighbor (ANN) searches (similarity search indexes for high-dimensionality data is very often approximate).You always want to build this indexafteryou have inserted the data, as the index needs to discover clusters in your data to be effective, and it does this only when first building the index.The index has a tunable parameter of the number of lists to use, and the code below shows the best practice for tuning this parameter. You also need to specify the distance measure used for indexing and ensure it matches the measure you use in your queries. In our case, we use the Cosine distance for querying below, and so we create our index withvector_cosine_ops.# Create an index on the data for faster retrieval

#calculate the index parameters according to best practices
num_lists = num_records / 1000
if num_lists < 10:
   num_lists = 10
if num_records > 1000000:
   num_lists = math.sqrt(num_records)

#use the cosine distance measure, which is what we'll later use for querying
cur.execute(f'CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops) WITH (lists = {num_lists});')
conn.commit()Part 3: Nearest Neighbor Search Using pgvectorGiven a user question, we’ll perform the following steps to use information stored in the vector database to answer their question using Retrieval Augmented Generation:Create an embedding vector for the user question.Use pgvector to perform a vector similarity search and retrieve theknearest neighbors to the question embedding from our embedding vectors representing the blog content. In our example, we’ll use k=3, finding the three most similar embedding vectors and associated content.Supply the content retrieved from the database as additional context to the model and ask it to perform a completion task to answer the user question.3.1 Define a question you want to answerFirst, we’ll define a sample question that a user might want to answer about the blog posts stored in the database.# Question about Timescale we want the model to answer
input = ""How is Timescale used in IoT?""Since Timescale ispopular for IoT sensor data, a user might want to learn specifics about how they can leverage it for that use case.3.2 Find the most relevant content in the databaseHere’s the function we use to find the three nearest neighbors to the user question. Note it uses pgvector’s<=>operator, which finds theCosine distance(also known as Cosine similarity) between two embedding vectors.# Helper function: Get top 3 most similar documents from the database
def get_top3_similar_docs(query_embedding, conn):
    embedding_array = np.array(query_embedding)
    # Register pgvector extension
    register_vector(conn)
    cur = conn.cursor()
    # Get the top 3 most similar documents using the KNN <=> operator
    cur.execute(""SELECT content FROM embeddings ORDER BY embedding <=> %s LIMIT 3"", (embedding_array,))
    top3_docs = cur.fetchall()
    return top3_docs3.3 Define helper functions to query OpenAIWe supply helper functions to create an embedding for the user question and to get a completion response from an OpenAI model. We use GPT-3.5, but you can use GPT-4 or any other model from OpenAI.We also specify a number of parameters, such as limits of the maximum number of tokens in the model response and model temperature, which controls the randomness of the model, which you can modify to your liking:# Helper function: get text completion from OpenAI API
# Note we're using the latest gpt-3.5-turbo-0613 model
def get_completion_from_messages(messages, model=""gpt-3.5-turbo-0613"", temperature=0, max_tokens=1000):
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=temperature, 
        max_tokens=max_tokens, 
    )
    return response.choices[0].message[""content""]

# Helper function: get embeddings for a text
def get_embeddings(text):
    response = openai.Embedding.create(
        model=""text-embedding-ada-002"",
        input = text.replace(""\n"","" "")
    )
    embedding = response['data'][0]['embedding']
    return embedding3.3 Putting it all togetherWe’ll define a function to process the user input by retrieving the most similar documents from our database and passing the user input, along with the relevant retrieved context to the OpenAI model to provide a completion response to.Note that we modify the system prompt as well in order to influence the tone of the model’s response.We pass to the model the content associated with the three most similar embeddings to the user input using the assistant role. You can also append the additional context to the user message.# Function to process input with retrieval of most similar documents from the database
def process_input_with_retrieval(user_input):
    delimiter = ""```""

    #Step 1: Get documents related to the user input from database
    related_docs = get_top3_similar_docs(get_embeddings(user_input), conn)

    # Step 2: Get completion from OpenAI API
    # Set system message to help set appropriate tone and context for model
    system_message = f""""""
    You are a friendly chatbot. \
    You can answer questions about timescaledb, its features and its use cases. \
    You respond in a concise, technically credible tone. \
    """"""

    # Prepare messages to pass to model
    # We use a delimiter to help the model understand the where the user_input starts and ends
    messages = [
        {""role"": ""system"", ""content"": system_message},
        {""role"": ""user"", ""content"": f""{delimiter}{user_input}{delimiter}""},
        {""role"": ""assistant"", ""content"": f""Relevant Timescale case studies information: \n {related_docs[0][0]} \n {related_docs[1][0]} {related_docs[2][0]}""}   
    ]

    final_response = get_completion_from_messages(messages)
    return final_responseLet’s see an example of the model’s output to our original input question:input = ""How is Timescale used in IoT?""
response = process_input_with_retrieval(input)
print(input)
print(response)Here’s the model’s response:User input: How is Timescale used in IoT?Model response:TimescaleDB is commonly used in IoT (Internet of Things) applications for storing and analyzing time-series data generated by IoT devices. IoT devices generate a large volume of data over time, such as sensor readings, device status updates, and alarm information. TimescaleDB provides a scalable and efficient solution for storing and querying this time-series data.With TimescaleDB, IoT applications can benefit from features such as automatic data partitioning, compression, and retention policies. These features allow for efficient storage and retrieval of time-series data, even as the data volume grows. TimescaleDB also supports SQL queries, making it easy to perform complex analytics and aggregations on the time-series data.In IoT applications, TimescaleDB can be used to monitor and analyze various aspects, such as environmental conditions (temperature, humidity), energy consumption, equipment performance, and predictive maintenance. It enables real-time monitoring, anomaly detection, and trend analysis, helping businesses make data-driven decisions and optimize their operations.Overall, TimescaleDB provides a reliable and scalable database solution for handling the high-volume time-series data generated by IoT devices, enabling businesses to unlock valuable insights and improve their IoT applications.We can also ask the model questions about specific documents in the database, in this case about specific Timescale users who have spoken about their experience in a blog post:input_2 = ""Tell me about Edeva and Hopara. How do they use Timescale?""
response_2 = process_input_with_retrieval(input_2)
print(input_2)
print(response_2)Here’s the model output for the input question above; notice how it uses specific details from the blog posts.User input: Tell me about Edeva and Hopara. How do they use Timescale?Model response:Edeva and Hopara are two companies that utilize TimescaleDB for their data storage and visualization needs.Edeva uses TimescaleDB as the main database in their smart city system. Their clients can control their IoT devices and view captured data, gaining insights from trends and historical data. They leverage TimescaleDB's continuous aggregations feature to speed up queries and improve dashboard performance.Hopara, on the other hand, provides a sophisticated visualization system that allows users to derive insights from various types of data. They use TimescaleDB to store real-time vibration data from sensor-tagged machines. Hopara's visualization system, powered by TimescaleDB, enables users to drill down into the data and identify vibration issues.Both Edeva and Hopara benefit from TimescaleDB's time-series functionality and its ability to handle large amounts of data efficiently.ConclusionRetrieval Augmented Generation (RAG) is a powerful method of building applications with LLMs that enable you to teach foundation models about things it was not originally trained on—like private documents or recently published information.We covered the basics of creating a chatbot to answer questions about a blog. We used the content from the Timescale Developer Q&A blog posts as an example to show how to create, store, and perform similarity search on OpenAI embeddings.We used PostgreSQL and pgvectoras our vector database to store and query the embeddings.✨Jupyter Notebook and Code:You can find all the code used in this tutorial in a Jupyter Notebook, as well as sample content and embeddings on the Timescale GitHub:timescale/vector-cookbook.And if you’re looking for a production PostgreSQL database for your vector workloads,try Timescale. It’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/postgresql-as-a-vector-database-create-store-and-query-openai-embeddings-with-pgvector/
2023-06-30T13:03:10.000Z,Nearest Neighbor Indexes: What Are ivfflat Indexes in pgvector and How Do They Work,"The rising popularity of ChatGPT, OpenAI, and applications of Large Language Models (LLMs) has brought the concept of approximate nearest neighbor search (ANN) to the forefront and sparked a renewed interest in vector databases due to the use of embeddings.Embeddingsare mathematical representations of phrases that capture the semantic meaning as a vector of numerical values.What makes this representation fascinating—and useful—is that phrases with similar meanings will have similar vector representations, meaning the distance between their respective vectors will be small. We recently discussed one application of these embeddings,Retrieval Augmented Generation—augmenting base LLMs with knowledge that it wasn’t trained on—but there are numerous other applications as well.One common application of embeddings is preciselysemantic similarity search. The basic concept behind this approach is that if I have a knowledge library consisting of various phrases and I receive a question from a user, I can locate the most relevant information in my library by finding the data that is most similar to the user's query.This is in contrast to lexical or full-text search, which only returns exact matches for the query. The remarkable aspect of this technique is that, since the embeddings represent the semantics of the phrase rather than its specific wording, I can find pertinent information even if it is expressed using completely different words!Semantic similarity search involves calculating an embedding for the user's question and then searching through my library to find the K most relevant items related to that question—these are the K items whose embeddings are closest to that of the question. However, when dealing with a large library, it becomes crucial to perform this search efficiently and swiftly. In the realm of vector databases, this problem is referred to as ""Finding the k nearest neighbors"" (KNN).This post discusses a method to enhance the speed of this search when utilizing PostgreSQL andpgvectorfor storing vector embeddings: theInverted File Flat (ivfflat)algorithm for approximate nearest neighbor search. We’ll cover why ivfflat is useful, how it works, and best practices for using it in pgvector for fast similarity search over embeddings vectors.Let’s go!Why Use the ivfflat Index in pgvector: The Curse of DimensionalitySearching for the k-nearest neighbors is not a novel problem for PostgreSQL.PostGIS, a PostgreSQL extension for handling location data, stores its data points as two-dimensional vectors (longitude and latitude). Locating nearby locations is a crucial query in that domain.PostGIS tackles this challenge by employing an index known as an R-Tree, which yields precise results for k-nearest neighbor queries. Similar techniques, such as KD-Trees and Ball Trees, are also employed for this type of search in other databases.However, there's a catch. These approaches cease to be effective when dealing with data larger than approximately 10 dimensions due to the ""curse of dimensionality."" Cue the ominous music! Essentially, as you add more dimensions, the available space increases exponentially, resulting in exponentially sparser data. This reduced density renders existing indexing techniques, like the aforementioned R-Tree, KD-Trees, and Ball Trees, which rely on partitioning the space, ineffective. (To learn more, I suggest these two videos:1,2).Given that embeddings often consist of more than a thousand dimensions—OpenAI’s are 1,536—new techniques had to be developed. There are no known exact algorithms for efficiently searching in such high-dimensional spaces. Nevertheless, there are excellentapproximatealgorithms that fall into the category of approximate nearest neighbor algorithms. Numerous such algorithms exist, but in this article, we will delve into the Inverted File Flat or ivfflat algorithm, which is provided by pgvector.How the ivfflat Index Works in pgvectorHow ivfflat divides the spaceTo gain an intuitive understanding of how ivfflat works, let's consider a set of vectors represented in a two-dimensional space as the following points:A set of vectors represented as points in two dimensionsIn the ivfflat algorithm, the first step involves applying k-means clustering to the vectors to find cluster centroids. In the case of the given vectors, let's assume we perform k-means clustering and identify four clusters with the following centroids.After k-means clustering, we identify four clusters indicated by the colored trianglesAfter computing the centroids, the next step is to assign each vector to its nearest centroid. This is accomplished by calculating the distance between the vector and each centroid and selecting the centroid with the smallest distance as the closest one. This process conceptually maps each point in space to the closest centroid based on proximity.By establishing this mapping, the space becomes divided into distinct regions surrounding each centroid (technically, this kind of division is called aVoronoi Diagram). Each region represents a cluster of vectors that exhibit similar characteristics or are close in semantic meaning.This division enables efficient organization and retrieval of approximate nearest neighbors during subsequent search operations, as vectors within the same region are likely to be more similar to each other than those in different regions.The process of assigning each vector to its closest centroid conceptually divides the space into distinct regions that surround each centroidBuilding the ivfflat index in pgvectorIvfflat proceeds to create aninverted indexthat maps each centroid to the set of vectors within the corresponding region. In pseudocode, the index can be represented as follows:inverted_index = {
  centroid_1: [vector_1, vector_2, ...],
  centroid_2: [vector_3, vector_4, ...],
  centroid_3: [vector_5, vector_6, ...],
  ...
}Here, each centroid serves as a key in the inverted index, and the corresponding value is a list of vectors that belong to the region associated with that centroid. This index structure allows for efficient retrieval of vectors in a region when performing similarity searches.Searching the ivfflat index in pgvectorLet's imagine we have a query for the nearest neighbors to a vector represented by a question mark, as shown below:We want to find nearest neighbors to the vector represented by the question markTo find the approximate nearest neighbors using ivfflat, the algorithm operates under the assumption that the nearest vectors will be located in the same region as the query vector. Based on this assumption, ivfflat employs the following steps:Calculate the distance between the query vector (red question mark) and each centroid in the index.Select the centroid with the smallest distance as the closest centroid to the query (the blue centroid in this example).Retrieve the vectors associated with the region corresponding to the closest centroid from the inverted index.Compute the distances between the query vector and each of the vectors in the retrieved set.Select the K vectors with the smallest distances as the approximate nearest neighbors to the query.The use of the index in ivfflat accelerates the search process by restricting the search to the region associated with the closest centroid. This results in a significant reduction in the number of vectors that need to be examined during the search. Specifically, if we have C clusters (centroids), on average, we can reduce the number of vectors to search by a factor of 1/C.Searching at the edgeThe assumption that the nearest vectors will be found in the same region as the query vector can introduce recall errors in ivfflat. Consider the following query:ivfflat can sometimes make errors when searching for nearest neighbors to a point at the edge of two regions of the vector spaceFrom visual inspection, it becomes apparent that one of the light-blue vectors is closer to the query vector than any of the dark-blue vectors, despite the query vector falling within the dark-blue region. This illustrates a potential error in assuming that the nearest vectors will always be found within the same region as the query vector.To mitigate this type of error, one approach is to search not only the region of the closest centroid but also the regions of the next closest R centroids. This approach expands the search scope and improves the chances of finding the true nearest neighbors.In pgvector, this functionality is implemented through the `probes` parameter, which specifies the number of centroids to consider during the search, as described below.Parameters for pgvector’s ivfflat ImplementationIn the implementation of ivfflat in pgvector, two key parameters are exposed: lists and probes.Lists parameter in pgvectorThelistsparameter determines the number of clusters created during index building (It’s called lists because each centroid has a list of vectors in its region). Increasing this parameter reduces the number of vectors in each list and results in smaller regions.It offers the following trade-offs to consider:Higherlistsvalue speeds up queries by reducing the search space during query time.However, it also decreases the region size, which can lead to more recall errors by excluding some points.Additionally, more distance comparisons are required to find the closest centroid during step one of the query process.Here are some recommendations for setting thelistsparameter:For datasets with less than one million rows, uselists =  rows / 1000.For datasets with more than one million rows, uselists = sqrt(rows).It is generally advisable to have at least 10 clusters.Probes parameter in pgvectorThe probes parameter is a query-time parameter that determines the number of regions to consider during a query. By default, only the region corresponding to the closest centroid is searched. By increasing the probes parameter, more regions can be searched to improve recall at the cost of query speed.The recommended value for the probes parameter isprobes = sqrt(lists).Using ivfflat in pgvectorCreating an indexWhen creating an index, it is advisable to have existing data in the table, as it will be utilized by k-means to derive the centroids of the clusters.The index in pgvector offers three different methods to calculate the distance between vectors: L2, inner product, and cosine. It is essential to select the same method for both the index creation and query operations. The following table illustrates the query operators and their corresponding index methods:Distance typeQuery operatorIndex methodL2 / Euclidean<->vector_l2_opsNegative Inner product<#>vector_ip_opsCosine<=>vector_cosine_opsNote: OpenAIrecommendscosine distance for its embeddings.To create an index in pgvector using ivfflat, you can use a statement using the following form:CREATE INDEX ON <table name> USING ivfflat (<column name> <index method>) WITH (lists = <lists parameter>);Replace<table name>with the name of your table and<column name>with the name of the column that contains the vector type.For example, if our table is namedembeddingsand our embedding vectors are in a column namedembedding, we can create an ivfflat index as follows:CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);Here’s a simple Python function that you can use to create an ivfflat index with the correct parameters forlistsandprobesas discussed above:def create_ivfflat_index(conn, table_name, column_name, query_operator=""<=>""): 
    index_method = ""invalid""
    if query_operator == ""<->"":
        index_method = ""vector_l2_ops""
    elif query_operator == ""<#>"":
        index_method = ""vector_ip_ops""
    elif query_operator == ""<=>"":
        index_method = ""vector_cosine_ops""
    else:
        raise ValueError(f""unrecognized operator {query_operator}"")

    with conn.cursor() as cur:
        cur.execute(f""SELECT COUNT(*) as cnt FROM {table_name};"")
        num_records = cur.fetchone()[0]

        num_lists = num_records / 1000
        if num_lists < 10:
            num_lists = 10
        if num_records > 1000000:
            num_lists = math.sqrt(num_records)

        cur.execute(f'CREATE INDEX ON {table_name} USING ivfflat ({column_name} {index_method}) WITH (lists = {num_lists});')
        conn.commit()QueryingAn index can be used whenever there is an ORDER BY of the formcolumn <query operator> <some pseudo-constant vector>along with a LIMIT k;Some examplesGet the closest two vectors to a constant vector:SELECT * FROM my_table ORDER BY embedding_column <=> '[1,2]' LIMIT 2;This is a common usage pattern in retrieval augmented generation using LLMs, where we find the embedding vectors that are closest in semantic meaning to the user’s query. In that case, the constant vector would be the embedding vector representing the user’s query.You can see an example of this in our guide tocreating, storing, and querying OpenAI embeddings with pgvector, where we use this Python function to find the three most similar documents to a given user query from our database:# Helper function: Get top 3 most similar documents from the database
def get_top3_similar_docs(query_embedding, conn):
    embedding_array = np.array(query_embedding)
    # Register pgvector extension
    register_vector(conn)
    cur = conn.cursor()
    # Get the top 3 most similar documents using the KNN <=> operator
    cur.execute(""SELECT content FROM embeddings ORDER BY embedding <=> %s LIMIT 3"", (embedding_array,))
    top3_docs = cur.fetchall()
    return top3_docsGet the closest vector to some row:SELECT * FROM my_table WHERE id != 1 ORDER BY embedding_column <=> (SELECT embedding_column FROM my_table WHERE id = 1) LIMIT 2;Tip:PostgreSQL's ability to use an index does not guarantee its usage! The cost-based planner evaluates query plans and may determine that a sequential scan or a different index is more efficient for a specific query. You can use the EXPLAIN command to see the chosen execution plan. To test the viability of using an index, you can modify planner costing parameters until you achieve the desired plan. For small datasets, settingenable_seqscan = 0can be especially advantageous for testing viability as it avoids sequential scans.To adjust the probes parameter, you can set theivfflat.probesvariable. For instance, to set it to '5', execute the following statement before running the query:SET ivfflat.probes = 5;Dealing with data changesAs your data evolves with inserts, updates, and deletes, the ivfflat index will be updated accordingly. New vectors will be added to the index, while no longer-used vectors will be removed.However, the clustering centroids will not be updated. Over time, this can result in a situation where the initial clustering, established during index creation, no longer accurately represents the data. This can be visualized as follows:As data gets inserted or deleted from the index, if the index is not rebuilt, the ivfflat index in pgvector can return incorrect approximate nearest neighbors due to clustering centroids no longer fitting the data wellTo address this issue, the only solution is to rebuild the index.Here are two important takeaways from this issue:Build the index once you have all representative data you want to reference in it, This is unlike most indexes which can be built on an empty table.It is advisable to periodically rebuild the index.When rebuilding the index, it is highly recommended to use the CONCURRENTLY option to avoid interfering with ongoing operations.Thus, to rebuild the index run the following in a cron job:REINDEX INDEX CONCURRENTLY <index name>;Summing It UpThe ivfflat algorithm in pgvector provides an efficient solution for approximate nearest neighbor search over high-dimensional data like embeddings. It works by clustering similar vectors into regions and building an inverted index to map each region to its vectors. This allows queries to focus on a subset of the data, enabling fast search. By tuning the lists and probes parameters, ivfflat can balance speed and accuracy for a dataset.Overall, ivfflat gives PostgreSQL the ability to perform fast semantic similarity search over complex data. With simple queries, applications can find the nearest neighbors to a query vector among millions of high-dimensional vectors. For natural language processing, information retrieval, and more, ivfflat is a compelling solution. By understanding how ivfflat divides the vector space into regions and builds its inverted index, you can optimize its performance for your needs and build powerful applications on top of it.✨Hands-on tutorials:Now that you know more about the ivfflat index in pgvector, get your hands dirty with using pgvector: Followour tutorialon creating, storing, and querying OpenAI embeddings using PostgreSQL as a vector database. Orlearn howto use pgvector as a vectorstore in LangChain.And if you’re looking for a production PostgreSQL database for your vector workloads,try Timescale. It’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/nearest-neighbor-indexes-what-are-ivfflat-indexes-in-pgvector-and-how-do-they-work/
2023-09-25T13:01:59.000Z,How We Made PostgreSQL a Better Vector Database,"Introducing Timescale Vector, PostgreSQL++ for production AI applications. Timescale Vector enhances pgvector with faster search, higher recall, and more efficient time-based filtering, making PostgreSQL your new go-to vector database. Timescale Vector is available today in early access on Timescale’s cloud data platform. Keep reading to learn why and how we built it. Then take it out for a ride:try Timescale Vector for free today, with a 90-day extended trial.The Rise of AI and Vector EmbeddingsAI is eating the world.The rise of large language models (LLMs) likeGPT-4,Llama 2, andClaude 2is driving explosive growth in AI applications. Every business is urgently exploring how to use LLM’s new, game-changing capabilities to better serve their customers—either by building new applications with AI or adding AI capabilities to their existing products.Vector datais at the heart of this Cambrian explosion of AI applications.In the field of LLMs, vector embeddings are mathematical representations of phrases that capture their semantic meaning as a vector of numerical values. Embeddings give you new superpowers for a key computing application:search.Embeddings enable search based on semantics (i.e., finding items in your database that are closest in meaning to the query) even if the words used are very different. They differ from lexicographical search, where you search for the use of similar words. (We covered vector embeddings in more detail in arecent blog post on finding nearest neighbors.)In addition to traditional semantic search applications, vector embeddings are crucial for making LLMs work with data the model wasn’t pre-trained on, like private information (e.g., company policies), or pre-trained on in-depth (e.g., your product documentation and capabilities), along with new information that’s emerged in the meantime (e.g., news, chat history).They are also at the core ofgenerative AItechniques like Retrieval Augmented Generation (RAG) to find relevant information to pass as context to an LLM. Other applications include everything from knowledge base search to classification and giving long-term memory to LLMs andAI agents.Vector Databases and the Paradox of ChoiceTo power next-generation AI systems, developers need to efficiently store and query vectors. There are a myriad of vector databases in the market, with new ones popping up seemingly every week. This leaves developers facing a paradox of choice. Do they adopt new, niche databases built specifically for vector data? Or do they use familiar, general-purpose databases, like PostgreSQL, extended with vector support?At Timescale, we have seen this dynamic before in other markets, namelytime-series data. Despite the existence of niche time-series databases, like InfluxDB and AWS Timestream, millions of developers chose to use a general-purpose database in PostgreSQL with TimescaleDB, an extension for time series.PostgreSQL's robustness, familiarity, and ecosystem outweighed switching to a completely new database. Additionally, data does not live in a silo, and the ability to join data of different modalities efficiently is often crucial for enabling interesting applications.With vector data, developers face a similar choice. And while developer needs for LLM applications are still being molded, we think PostgreSQL will come out on top and become the foundational database for complex, production-grade AI applications, just as it has been in applications over the past decades.Promises and Pitfalls of Specialized Vector DatabasesNiche databases for vector data like Pinecone, Weaviate, Qdrant, and Zilliz benefited from the explosion of interest in AI applications. They come purpose-built for storing and querying vector data at scale—with unique features like indexes for Approximate Nearest Neighbor (ANN) search and hybrid search. But as developers started using them for their AI applications, the significant downsides of building with these databases became clear:Operational complexity:Continued maintenance of a separate database just for vector data adds another layer of operational overhead, requiring teams to duplicate, synchronize, and keep track of data across multiple systems. Not to mention backups, high availability, and monitoring.Learning curve:Engineering teams lose time learning a new query language, system internals, APIs, and optimization techniques.Reliability:Building a robust database from scratch is a monumental challenge. Most niche vector databases are unproven, nascent technologies with questionable long-term stability and reliability.In the words of one developer we interviewed:""Postgres is more production-ready, more configurable, and more transparent in its operation than almost any other vector store.""-Software Engineer at LegalTech startupPostgreSQL as a Vector DatabasePostgreSQL is the most loved database in the world, according to theStack Overflow 2023 Developer Survey. And for a good reason: it’s been battle-hardened by production use for over three decades, it’s robust and reliable, and it has a rich ecosystem of tools, drivers, and connectors.PostgreSQL is the most loved database in the world according to the Stack Overflow 2023 Developer SurveyAnd amidst the sea of new, niche vector databases, there is an undeniable appetite for using PostgreSQL as a vector database—look no further than the numeroustutorials, integrations, and tweets aboutpgvector, the open-source PostgreSQL extension for vector data.One of the many great features of PostgreSQL is that it is designed to be extensible. These “PostgreSQL extensions” add extra functionality without slowing down or adding complexity to core development and maintenance. It’s what we leveraged for building TimescaleDB and how pgvector came about as well.While pgvector is a wonderful extension (and is offered as part of Timescale Vector), it is just one piece of the puzzle in providing a production-grade experience for AI application developers on PostgreSQL. After speaking with numerous developers at nimble startups and established industry giants, we saw the need to enhance pgvector to cater to the performance and operational needs of developers building AI applications.Enter Timescale Vector: PostgreSQL++ for AI ApplicationsToday, we launch Timescale Vector to enable you, the developer, to build production AI applications at scale with PostgreSQL.Timescale Vector speeds up ANN search on millions of vectors, enhancing pgvector with a state-of-the-art ANN index inspired by theDiskANNalgorithm, in addition to offering pgvector’s hierarchical navigable small world (HNSW) and inverted file (IVFFlat) indexing algorithms. Our benchmarking shows that Timescale Vector achieves 243% faster search speed at 99% recall than Weaviate, a specialized vector database, and between 39.39% and 363.48% faster search speed than previously best-in-class PostgreSQL search indexes (pgvector’s HNSW and pg_embedding, respectively) on a dataset of one million OpenAI embeddings.Timescale Vector’s new index outperforms specialized vector database Weaviate by 243% and all existing PostgreSQL index types when performing approximate nearest neighbor searches at 99% recall on one million OpenAI vector embeddingsTimescale Vector optimizes hybrid time-based vector search,leveraging the automatic time-based partitioning and indexing ofTimescale’s hypertablesto efficiently find recent embeddings, constrain vector search by a time range or document age, and store and retrieve LLM response and chat history with ease.# Create client object
TIME_PARTITION_INTERVAL = timedelta(days=7)
vec = client.Async(TIMESCALE_SERVICE_URL,
                  TABLE_NAME, 
                  EMBEDDING_DIMENSIONS,
                  time_partition_interval=TIME_PARTITION_INTERVAL)

# create table
await vec.create_tables()

# similarity search with time-based filtering
records_time_filtered = await vec.search(query_embedding,
                       limit=3,
                       uuid_time_filter=client.UUIDTimeRange(start_date, end_date))Timescale Vector simplifies the AI application stack, giving you a single place for the vector embeddings, relational data, time series, and event data that powers your next-generation AI applications. You don’t need to manage yet another piece of infrastructure, and it minimizes the operational complexity of data duplication, synchronization, and keeping track of updates across multiple systems. Because Timescale Vector is still PostgreSQL, it inherits its 30+ years of battle testing, robustness, and reliability, giving you more peace of mind about your database of choice for data that’s critical to great user experience.Timescale Vector simplifies handling metadata and multi-attribute filtering,so you can leverage all PostgreSQL data types to store and filter metadata, JOIN vector search results with relational data for more contextually relevant responses, and write full SQL relational queries incorporating vector embeddings. In future releases, Timescale Vector will also further optimize rich multi-attribute filtering, enabling even faster similarity searches when filtering on metadata. (We are actively seeking feedback on use cases for complex filtering, so pleasereach outif you have such a use case.)In addition to unique capabilities for handling vector data at scale, Timescale Vector sits atop Timescale’s production-grade cloud PostgreSQL platform, complete with:Flexible and transparent pricing:Timescale decouples compute and storage, enabling you to scale resources independently as you grow. Only pay for what you use withusage-based storage pricing.Production PostgreSQL platform features: Push to prod with the confidence of automatic backups, failover, and high availability. Use read-replicas to scale query load. Take advantage of one-click database forking for testing new embeddings and LLMs.Enterprise-grade security and data privacy:Timescale is SOC2 Type II- and GDPR-compliant and keeps your data secure and private with data encryption at rest and in motion, VPC peering to your Amazon VPC, secure backups, and multi-factor authentication.Free support from PostgreSQL experts:Timescale runs production PostgreSQL databases for more than a thousand customers. And we offer consultative support to guide you as you grow at no extra cost.💡Think of Timescale Vector as a complement, not a replacement for pgvector. Timescale Vector uses the same vector data type as pgvector, offering all its other capabilities (like HNSW indexes). This gives you the flexibility to create different index types suited to your needs while leveraging the new capabilities described above to build your AI applications.Developers are excitedvia GIPHYSince we opened up a waitlist for Timescale Vector, we have spoken to numerous developers at companies large and small about their AI applications and use of vector data. We want to publicly thank each and every one of them for informing our roadmap and helping shape the initial product direction.Here’s what they had to say about Timescale Vector:“The simplicity and scalability of Timescale Vector's integrated approach to use Postgres as a time-series and vector database allows a startup like us to bring an AI product to market much faster.Choosing TimescaleDB was one of the best technical decisions we made, and we are excited to use Timescale Vector.” – Nicolas Bream, CEO atPolyPerception.“Being able to utilize conditions on vectors for similarity, alongside traditional time and value conditions simplifies our data pipelinesand allows us to lean on the strengths of PostgreSQL for searching large datasets very quickly. Timescale Vector allows us to efficiently search our system for news related to assets, minimizing our reliance on tagging by our sources.”– Web Begole, CTO atMarketReader.“Using Timescale Vector allows us to easily combine PostgreSQL’s classic database features with storage of vector embeddings for Retrieval Augmented Generation (RAG). Timescale’s easy-to-use cloud platform and good support keep our team focused on imaging solutions to solve customer pains not on building infrastructure.""- Alexis de Saint Jean, Innovation Director atBlueway Software.“Postgres is obviously battle-tested in a way that other vector stores aren't.Timescale Vector benefits from Timescale’s experience offering cloud PostgreSQL,and our organization is excited to use it to confidently and quickly get our LLM-based features into production.” - Software Engineer at a LegalTech company.How to access Timescale VectorTimescale Vector is available today in early access onTimescale, the PostgreSQL cloud platform, for new and existing customers. The easiest way to access Timescale Vector is via theTimescale Vector Python client library, which offers a simple way to integrate PostgreSQL and Timescale Vector into your AI applications.To get started, create a new database on Timescale, download the.envfile with your database credentials, and run:pip install timescale-vectorThen,see the Timescale Vector docsfor instructions or learn the key features of Timescale Vector by followingthis tutorial.Try Timescale Vector for free today on Timescale.Three-month free trial for new customers:We’re giving new Timescale customers anextended 90-day trial. This makes it easy to test and develop your applications with Timescale Vector, as you won’t be charged for any cloud PostgreSQL databases you spin up during your trial period.Special early access pricing:Because we can’t wait for you to try Timescale Vector, during the early access period, Timescale Vector will be free to use for all Timescale new and existing customers. We’ll disclose our final pricing for Timescale Vector in the coming weeks. Stay tuned for details.Why early access❓Our goal is to help developers power production AI applications, and we have a high bar for the quality of such a critical piece of infrastructure as a database. So, during this early access period, we’re inviting developers to test and give feedback on Timescale Vector so that they can shape the future of the product as we continue to improve, building up to a general availability release.To celebrate the early access release of Timescale Vector, we’re launching something new to help you build AI applications every day this weekThis is the first of several exciting announcements as part of Timescale AI Week. You can find something new to help you build AI applications every day attimescale.com/ai. Read on to learn more about why we built Timescale Vector, our new DiskANN-inspired index, and how it performs against alternatives.Speeding Up Vector Search in PostgreSQL With a DiskANN-Inspired IndexAs part of Timescale Vector, we are excited to introduce a new indexing technique to PostgreSQL that improves vector search speed and accuracy.ANN search on vector data is a bustling field of research. Fresh methodologies and algorithms emerge annually. For a glimpse into this evolution, one can peruse the proceedings ofNeurIPS 2022.It is exciting to note that PostgreSQL-based vector stores, in particular, are at the forefront of this evolution. We previously discussed how to usepgvector’s IVFFlat indextype to speed up search queries for ANN queries, but it has performance limitations at the scale of hundreds of thousands of vectors. Since then, teams across the PostgreSQL community are implementing more index types based on the latest research.For instance, pgvector just releasedsupport for HNSW(a graph-based index type for vector data). Neon’spg_embeddingwas also recently released with support for HNSW.And as part of Timescale Vector, we’re announcing support for a new graph-based index,tsvdrawing inspiration from Microsoft’sDiskANN. Its graph-construction approach differs from HNSW, allowing for different trade-offs and unique advantages.You can create a timescale-vector index on the column containing your vector embeddings using our Python client as follows:# Create a timescale vector (DiskANN) search index on the embedding column
await vec.create_embedding_index(client.TimescaleVectorIndex())And using SQL to create the timescale-vector index looks like:CREATE INDEX ON <table name> USING tsv(<vector_column_name>);We'll delve deeper into why the algorithms used by this index are particularly advantageous in PostgreSQL. We are staunch proponents of offering diverse options to the PostgreSQL community, leading us to design an index distinct from those already available in the ecosystem.Our decision to house our index in a standalone extension from pgvector was influenced by our use of Rust (for context, pgvector is written in C). Yet, we aimed to simplify its adoption by avoiding introducing a new vector data type. Hence, our index works with the vector data type provided by pgvector. This allows users to easily experiment with different index types by creating different indexes on the same table contents.Thetimescale-vectorDiskANN-inspired index offers several compelling benefits:Optimized for SSDsMuch of the research on graph-based ANN algorithms focuses on graphs designed for in-memory use. In contrast, DiskANN is crafted with SSD optimization in mind. Its unique on-disk layout of graph nodes clusters each node vector with its neighboring links.This ensures that each node's visit during a search preloads the data required for the subsequent step, aligning seamlessly with Postgres' page caching system to minimize SSD seeks. The absence of graph layers (unlike HNSW) further augments the cache's efficiency, ensuring that only the most frequently accessed nodes are retained in memory.Quantization optimization compatible with PostgreSQLDiskANN gives users the option to quantize vectors within the index. This process reduces the vector size, consequently shrinking the index size significantly and expediting searches.While this might entail a marginal decline in query accuracy, PostgreSQL already stores the full-scale vectors in theheap-table. This allows for correcting the diminished accuracy from the indexed data using heap data, refining the search results. For instance, when searching forkitems, the index can be prompted for2kitems, which can then be re-ranked using heap data to yield the closestkresults.In our benchmarking, we saw a 10x index size reduction by enabling product quantization, reducing the index size from 7.92 GB to just 790 MB.The timescale-vector index has the option to quantize vectors within the index, leading to 10x space savings compared to not using product quantization, and compared to the pgvector HNSW index sizeTo use thetimescale-vectorindex with product quantization (PQ) enabled, you can create the index as follows:CREATE INDEX ON <table name> USING tsv(<column name>) WITH (use_pq=true);Filtering additional dimensionsFrequently referred to as hybrid search, this feature enables the identification of the k-nearest neighbors to a specific query vector while adhering to certain criteria on another column. For instance, vectors could be classified based on their source (such as internal documentation, external references, blog entries, and forums). A query from an external client might exclude internal documents, whereas others might limit their search to only blog entries and forums.Timescale Vector already supports hybrid search, but the current version does not optimize the search in all cases. The exciting thing is that the DiskANN authors have outlined a blueprint for a filtered DiskANN, and we plan to add this optimization before Timescale Vector reaches GA.Performing hybrid search using the Timescale Vector Python client can be done as follows:records_filtered = await vec.search(query_embedding, filter={""author"": ""Walter Isaacson""})In sum, the introduction of the new timescale-vector index type inspired by DiskANN benefits developers in the following ways:It increases the speed (both latency and throughput) of queries.It improves the accuracy of results.It decreases resource requirements.It supports more types of queries (in particular, queries that filter on multiple attributes, not just vectors).Benchmarking ANN Search: Methodology and ResultsNow for some numbers. Let’s look at how Timescale Vector’s approximate nearest neighbor search performance compares to a specialized vector database, in this case,Weaviate, and existing PostgreSQL search index algorithms.What we comparedWe compared the approximate nearest neighbor search performance of the following index types:Timescale Vector’s DiskANN-inspired indexWeaviate’s HNSW search indexPgvector’s recently released HNSW indexNeon’s pg_embedding HNSW indexPgvector’s IVFFlat indexMetrics trackedWe tracked a variety of metrics provided by theANN benchmarking suite, but the ones we’ll cover in-depth are the following:Queries per second: how many approximate nearest neighbor search queries per second can be processed using the index.Accuracy: what percentage of the returned approximate nearest neighbors are actually the true nearest neighbors of the given query vector? This is also known as the recall. (Measured as a value between 0 and 1, with 1 being 100%).Index size: how large is the index, which we convert to a value in gigabytes (GB).Index build time: how much time does it take to build the index? This is measured in seconds, but we convert it to minutes in the table below for easier understanding.Note: pgvector is packaged as part of Timescale Vector, so developers have the flexibility to choose the right index type, whether it’s DiskANN, HNSW, or IVFFlat—or opt to use exact KNN search for their use case.Benchmarking toolWe used the popular set of tools fromANN Benchmarksto benchmark the performance of the different algorithms against each other on the same dataset. We do experiments using two modes: a parallel (--batch) mode, where multiple queries are executed at once, and a single-threaded mode, where queries are executed one at a time.DatasetWe benchmarked the performance of the various algorithms on a dataset ofone million OpenAI vector embeddings. Each vector has 1,536 dimensions and was created using OpenAI’stext-embedding-ada-002 embedding model. (Fun fact: the dataset is based on embedding content from Wikipedia articles). Shout-out toKumar Shivendufor making that dataset easily available and adding it to ANN benchmarks.Machine configurationWe used the following setup:Versions: we tested the initial version of Timescale Vector, pgvector 0.5.0 for both HNSW and IVFFlat, pg_embedding 0.3.5, and Weaviate 1.19.0-beta.1.Instance details: both the benchmark client and the database server ran on the same AWS EC2 virtual machines running Ubuntu 22.04.2 LTS with 8 vCPU,  32 GB Memory, and a 300 GB EBS volume. Note that this configuration is available on the Timescale cloud platform.Algorithm parametersWe varied the following parameters over different runs to test their impact on the performance vs. accuracy trade-off. We tried to use parameters already present inANN Benchmarks(often suggested by the index creators themselves) whenever possible.Timescale Vector (DiskANN):num_neighborsvaried between 50 and 64, which sets the maximum number of neighbors per node. Defaults to 50. Higher values increase accuracy but make the graph traversal slower.search_list_sizewas varied between 10 and 100: this is the S parameter used in the greedy search algorithm used during construction. Defaults to 100. Higher values improve graph quality at the cost of slower index builds.max_alphavaried between 1.0 and 1.2:max_alphais the alpha parameter. Defaults to 1.0. Higher values improve graph quality at the cost of slower index builds.query_search_list_sizevaried between 10 and 250: this is the number of additional candidates considered during the graph search at query time. Defaults to 100. Higher values improve query accuracy while making the query slower.We also compared Timescale Vector’s index performance with and without product quantization enabled. When PQ was enabled, we setnum_clustersto 256.num_clusterssets the number of clusters (and centroids) used for every segment’s quantizer.  Higher values improve accuracy at the cost of slower index builds and queries.If you want to dive deeper into the timescale vector indexing algorithm, its parameters, and how it works, we discuss it in more depth in the “Under the Hood” section.pgvector HNSW:mvaried between 12 and 36:mrepresents the maximum number of connections per layer. Think of these connections as edges created for each node during graph construction. Increasingmincreases accuracy but also increases index build time and size.ef_constructionvaried between 40 and 128:ef_constructionrepresents the size of the dynamic candidate list for constructing the graph. It influences the trade-off between index quality and construction speed. Increasingef_constructionenables more accurate search results at the expense of lengthier index build times.ef_searchvaried between 10 and 600:ef_searchrepresents the size of the dynamic candidate list for search. Increasingef_searchincreases accuracy at the expense of speed.Weaviate HNSW:maxConnectionsvaried between 8 and 72. This parameter is explained undermin the pgvector section above.efConstructionvaried between 64 and 512. This parameter is explained byef_constructionin the pgvector section above.efvaried between 16 and 768. This parameter is explained byef_searchin the pgvector section above.pg_embedding HNSW:dimswas set to 1,536, the number of dimensions in our data.mvaried between 12 and 36.efconstructionvaried between 40 and 128.efsearchvaried between 10 and 600.pgvector IVFFlat:listsvaried from 100 to 4000:listsrepresents the number of clusters created during index building (It’s called lists because each centroid has a list of vectors in its region). Increasinglistsreduces the number of vectors in each list and results in smaller regions, but it can introduce more errors as some points are excluded.probesvaried from 1 to 100:probesrepresents the number of regions to consider during a query. Increasingprobesmeans more regions can be searched and improves accuracy at the expense of query speed.For more on IVFFlat parameters, see thisIVFFlat explainer blog post.In general, we want to evaluate the accuracy versus speed trade-off between the various approximate nearest neighbor algorithms.Benchmarking ResultsQueries per secondSingle-threaded experimentFirst, let’s take a look at the results of the single-threaded mode experiment, where queries are executed one at a time.The single-threaded experiment shows that Timescale Vector’s new index type inspired by DiskANN comes out on top in terms of query performance at 99% accuracy. It outperforms Weaviate, a specialized vector database, by 122.05%. It also outperforms all existing PostgreSQL ANN indexes, beating pgvector’s HNSW algorithm by 29.24%, pg_embedding by 257.56%, and pgvector’s IVFFLAT by 1,383.26%.Note that if you’re willing to lower accuracy to less than 99%, say ~95%, you can get much faster query throughput. For example, at 96% accuracy, Timescale Vector’s DiskANN-inspired index can process 425 queries per second, and pgvector HNSW can process 376 queries per second.We achieved the above results with the following parameters:Timescale Vector: num_neighbors=64, search_list_size=100, max_alpha=1.0 query_search_list_size=100.pgvector HNSW: m=36, ef_construction=128, ef_search=100.Weaviate: ef=48, maxConnections=40, efConstruction=256.pg_embedding: m=24, efconstruction=40, query_ef_search=600.pgvector IVFFlat: lists=2000, probes=100.Next, let’s look at the results from the parallel (--batch) mode experiments, where multiple queries are executed at once.Multi-threaded experimentIt is important to ensure that search algorithms scale with the number of CPUs, so let’s look at the results comparing the queries per second of the different vector search indexes when they are run in parallel:Timescale Vector’s new index outperforms specialized vector database Weaviate by 243% and all existing PostgreSQL index types when performing approximate nearest neighbor searches at 99% recall on one million OpenAI vector embeddingsKey TakeawaysThe multi-threaded experiment shows that Timescale Vector’s new index again comes out on top for queries per second at 99% accuracy:Timescale Vector outperforms Weaviate, a specialized vector database, by 243.77%.Timescale Vector also outperforms all existing PostgreSQL indexes, beating pgvector’s HNSW by 39.39%, pg_embedding’s HNSW by 362.48%, and pgvector’s IVFFlat by 1,590%.Apart from Weaviate, we didn’t test other specialized databases, but we can get an idea of how this result extends to other specialized vector databases like Qdrant by doing a loose comparison toQdrant benchmarks, which were also performed using ANN benchmarks on the same dataset of one million OpenAI embeddings, and used the same machine spec (8 CPU, 32 GB RAM) and experiment method (batch queries).With 1,252.20 queries per second at 99% recall, Timescale Vector surpassesQdrant’s 354 queries per secondby more than 250%.In the future, we plan to test Timescale Vector against more specialized vector databases and benchmark the results.While not depicted on the graph above, Timescale Vector with PQ enabled reaches 752 queries per second at 99% recall, giving it 106.61% better results than Weaviate, and 177.94% better results than pg_embedding. In terms of overall performance, it would rank 3rd in our benchmark behind Timescale Vector without PQ enabled and pgvector HNSW. These results are all the more impressive considering the index size reduction that enabling PQ yields, as discussed below.Another interesting result is that pgvector’s HNSW implementation is 232% more performant as Neon's pg_embedding’s HNSW implementation and 146% more performant as Weaviate’s HNSW implementation when running queries in parallel. Finally, the graph-based methods (DiskANN or HNSW) all outperform the inverted file method (IVFFlat) by a wide margin.The good news is—because Timescale Vector includes all of pgvector’s indexes (and other features)—you can create and test both Timescale Vector’s DiskANN and pgvector’s HNSW indexes on your data. Flexibility for the win!We achieved the above results for the multi-threaded experiment with the following parameters:Timescale Vector: num_neighbors=50, search_list_size=100, max_alpha=1.0 query_search_list_size=100Timescale Vector with PQ: num_neighbors=64, search_list_size=100, max_alpha=1.0, num_clusters=256 query_search_list_size=100pgvector HNSW: m=36, ef_construction=128, ef_search=100Weaviate: ef=64, maxConnections=32, efConstruction=256pg_embedding: m=24, efconstruction=56, query_ef_search=250pgvector IVFFlat: lists=2000, probes=100Index sizeLet’s look at the index size for the respective runs that yielded the best performance in the multi-threaded experiment:Timescale Vector with Product Quantization enables achieves 10x smaller index size than pgvector HNSW. Note the Weaviate index size was not correctly reported via ANN Benchmarks and so is not reflected on the graph above.Timescale Vector without PQ comes in at 7.9 GB, as does pgvector HNSW, pg_embedding HNSW and pg_vector IVFFlat. However, by enabling PQ, Timescale Vector decreases its space usage for the index by 10x, leading to an index size of just 790 MB.Note that we uncovered a bug in ANN Benchmarks which incorrectly reported the Weaviate index size in a previous version of this post. ANN Benchmarks reports the index size aspsutil.Process().memory_info().rss / 1024by default and that setting is not overridden for Weaviate. We've not reflected Weaviate on the graph above for that reason.Index build timeNow, we compare the build times for the above results:pgvector’s IVFFlat index comes out on top for index build time thanks to its simpler construction. Looking at the most performant indexes in terms of query performance, Timescale Vector is 75.41% faster to build than pgvector’s HNSW in this case. And Timescale Vector with PQ is 59.91% faster to build than pgvector HNSW.Many developers we’ve spoken to are willing to trade off index build time with higher query speed, as it's an upfront time cost (since the index is only built once) rather than a recurring time cost and could potentially affect their users’ experience. Moreover, the Timescale Vector team is working on decreasing the index build time by using parallelism in upcoming releases.If you’re looking for a fun weekend project, we encourage you to replicate these with these parameters to verify the results for yourself using the ANN benchmarks suite.Under the Hood: How Timescale Vector’s New Index WorksTimescale Vector’s new DiskANN-inspired index is based on a graph algorithm. The diagram above illustrates a simple graph and traversal to find the nearest neighbor to a given query vectorGraph-based indexes usually work like this: each vector in a database becomes a vertex in a graph, which is linked to other vertices. When a query arises, the graph is greedily traversed from a starting point, leading us to the nearest neighboring node to the queried vector.The example above shows that following the path with arrows from the graph entry point leads to the nearest neighbor.Let's break down how the timescale-vector index works, focusing on the search and construction algorithms. We'll begin with the search, as it plays a pivotal role in the construction process.Search algorithmThe objective of the greedy search is to pinpoint the nodes closest to a specific query vector. This algorithm operates in a loop until it reaches a fixed point. Within each loop iteration, the following happens:A vertex is ""visited,"" meaning its distance and that of its neighboring vertices to the query vector are calculated and inserted into a list of candidates ranked by proximity.The nearest yet-to-be-visited candidate on this ranked list is the one that is visited.The loop concludes when the top 'S' candidates, in terms of proximity, have all been visited, ensuring that no neighbor of these 'S' candidates is nearer to the query than the candidates themselves.When searching for k-nearest neighbors, you set 'S' as 'K' plus an added buffer B. The larger this buffer, the greater the number of candidates required for loop completion, rendering the search process slower but ensuring higher accuracy. At query time, you set the buffer size using thetsv.query_search_list_sizeGUC.Construction algorithmThe construction phase commences with a singular starting vertex, chosen arbitrarily. The graph is built iteratively, adding one node at a time. For each added node, the following happens:A search is initiated for that yet-to-be-included vertex, noting the vertices visited during this search.These visited vertices are designated as neighbors of the newly integrated vertex.Connections between neighbors are established in both directions.As construction progresses, edges in the graph are pruned using an “alpha-aware RNG*” property that states that a neighbor of V is pruned if it is closer (by a factor alpha) to another neighbor of V than to V itself.This is a brief overview of the mechanism of the DiskANN approach. In a future blog post, we’ll dive into the background of how graph indexes work and how the DiskANN approach differs from other algorithms in the space.Enabling Efficient Time-based Filtering in Vector SearchA key use case that Timescale Vector uniquely enables is efficient time-based vector search. In many AI applications, time is an important metadata component for vector embeddings.Documents, content, images, and other sources of embeddings have a time associated with them, and that time is commonly used as a filter for increasing the relevance of embeddings in an ANN search. Time-based vector search enables the retrieval of vectors that are not just similar but also pertinent to a specific time frame, enriching the quality and applicability of search results.Time-based vector functionality is helpful for AI applications in the following ways:Chat history:storing and retrieving LLM response history, for example, chat history for chatbots.Finding recent embeddings:finding the most recent embeddings similar to a query vector. For instance, the most recent news, documents, or social media posts related to elections.Search within a time range:constraining similarity search to only vectors within a relevant time range. For example, asking time-based questions about a knowledge base (“What new features were added between January and March 2023?”) or conducting an image similarity search on vectors within a specified time range.Anomaly detection: you often want to find anomalous vectors within a specified time range.Yet, traditionally, searching by two components’ “similarity” and “time” is challenging for approximate nearest neighbor indexes and makes the similarity-search index less effective, as similarity search alone can often return results not in the time window of interest.One approach to solving this is partitioning the data by time and creating ANN indexes on each partition individually. Then, during the search, you can do the following:Step 1: filter our partitions that don’t match the time predicate.Step 2: perform the similarity search on all matching partitions.Step 3: combine all the results from each partition in step 2, rerank, and filter out results by time.To solve this problem, Timescale Vector leverages TimescaleDB’s hypertables, which automatically partition vectors and associated metadata by a timestamp. This enables efficient querying on vectors by both similarity to a query vector and time, as partitions not in the time window of the query are ignored, making the search a lot more efficient by filtering out whole swaths of data in one go.Hear from Web Begole, CTO atMarketReader, who aims to use Timescale Vector’s time-based search functionality to help find news stories related to stock market movements more efficiently:“Being able to utilize conditions on vectors for similarity, alongside traditional time and value conditions, simplifies our data pipelinesand allows us to lean on the strengths of PostgreSQL for searching large datasets very quickly. Timescale Vector allows us to efficiently search our system for news related to assets, minimizing our reliance on tagging by our sources.”Here’s an example of using semantic search with time filters using theTimescale Vector Python client:# define search query
query_string = ""What's new with PostgreSQL 16?""
query_embedding = get_embeddings(query_string)


# Time filter variables for query
start_date = datetime(2023, 8, 1, 22, 10, 35) # Start date = 1 August 2023, 22:10:35
end_date = datetime(2023, 8, 30, 22, 10, 35) # End date = 30 August 2023, 22:10:35


# Similarity search with time filter
records_time_filtered = await vec.search(query_embedding,
                       limit=3,
                       uuid_time_filter=client.UUIDTimeRange(start_date, end_date))Try Timescale Vector todayTimescale Vector is available today in early access on Timescale, the PostgreSQL++ cloud platform, for new and existing customers.Try Timescale Vector for free today on Timescale.Documentation:Read the Timescale Vector docs for instructions on how to get started.Tutorial:Learn how to use Timescale Vector to perform semantic search, hybrid search, and time-based semantic search, and how to create indexes to speed up your queries.And a reminder:Three-months free for new customers:We’re giving new Timescale customers anextended 90-day trial. This makes it easy to test and develop your applications with Timescale Vector, as you won’t be charged for any cloud PostgreSQL databases you spin up during your trial period.Special early access pricing:Because we can’t wait for you to try Timescale Vector, during the early access period, Timescale Vector will be free to use for all Timescale new and existing customers. We’ll disclose our final pricing for Timescale Vector in the coming weeks.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-we-made-postgresql-the-best-vector-database/
2022-12-08T19:34:59.000Z,The Ultimate Guide to Time-Series Analysis (With Examples and Applications),"What Is Time-Series Analysis?Time-series analysis comprises the process and mathematical set of tools used for looking into time-series data to learn not onlywhathappened but alsowhenandwhyit happened, as well as what is most likely to happen in the future.‌‌Use Cases for Time-Series AnalysisThe “time” element in time-series data means that the data is ordered by time. In this type of data, each entry is preceded and followed by another and has a timestamp that determines the order of the data. Check out our earlier blog postto learn more and see examples of time-series data.A typical example of time-series data is stock prices or a stock market index. However, even if you’re not into financial and algorithmic trading, you probably interact daily with time-series data.When you drive your car through a digital toll or your smartphone tells you to walk more or that it will rain, time—series data is part of these interactions.To illustrate this in more detail, let’s look at the example of health apps.Real-World ExamplesIf you open a health app on your phone, you will see all sorts of categories, from step count to noise level or heart rate. By clicking on “show all data” in any of these categories, you will get an almost endless scroll (depending on when you bought the phone) of step counts, which were timestamped when the data was sampled. ‌‌This is the raw data of the step count time series. Remember, this is just one of many parameters sampled by your smartphone or smartwatch. While many parameters don’t mean much to most people (yes, I’m looking at you, heart rate variability), when combined with other data, these parameters can give you estimations on overall quantifiers, such as cardio fitness. ‌‌To achieve this, you need to connect the time-series data into one large dataset with two identifying variables—time and type of measurement. This is calledpanel data. Separating it by type gives you multiple time series, while picking one particular point in time gives you a snapshot of everything about your health at a specific moment, like what was happening at 7:45 a.m.Why Should You Use It?Now that you’re more familiar with time-series data, you may wonder what to do with it and why you should care. So far, we’ve been mostly just reading off data—how many steps did I take yesterday? Is my heart rate okay? Did it rain yesterday?‌‌But time-series analysis can help us answer more complex or future-related questions, such as forecasting. When did I stop walking and caught the bus yesterday? Is exercise making my heart stronger? Will it rain tomorrow? ‌‌To answer these, we need more than just reading the step counter at 7:45 a.m.—we need time-series analysis. Time-series analysis happens when we consider part or the entire time series to see the “bigger picture.”  We can do this manually in straightforward cases: for example,by looking at the graphthat shows the days when you took more than 10,000 steps this month. ‌‌But if you wanted to know how often this occurs or on which days, that would be significantly more tedious to do by hand. Very quickly, we bump into problems that are too complex to tackle without using a computer, and once we have opened that door, a seemingly endless stream of opportunities emerges. We can analyze everything, from ourselves to our business, and make them far more efficient and productive than ever.‌‌Time-series componentsLet’s go back to our health app example. One thing you may see immediately, just by looking at a time-series analysis chart, is whether your stats are trending upward or downward. That indicates whether your stats are generally improving or not. By ignoring the short-term variations, it's easier to see if the values rise or decline within a given time range. This is the first of the four components of a time series:Trend: as explained above, a long-term movement of the time series, such as the decreasing average heart rate of workouts as a person gets fitter.Seasonality: regular periodic occurrences within a time interval smaller than a year (e.g., higher step count in spring and autumn because it’s not too cold or too hot for long walks).Cyclicity: repeated fluctuations around the trend that are longer in duration than irregularities but shorter than what would constitute a trend. In our walking example, this would be a one-week sightseeing holiday every four to five months.Irregularity: short-term irregular fluctuations or noise, such as a gap in the sampling of the pedometer or an active team-building day during the workweek.Example data in black with the trend in red and the trend with seasonality in blue, cyclic fluctuations, and two large irregularities complete the decomposition (source: author)Limitations of time-series analysisIf you’re performing time-series analysis, it can be helpful to decompose it into these four elements to explain results and make predictions.Trendandseasonalityare deterministic, whereascyclicityandirregularitiesare not.Therefore, you first need to eliminate random events to know what can be understood and predicted. Nothing is perfect, and to be able to capture the full power of time-series analysis without abusing the technique and obtaining incorrect results and conclusions, it’s essential to address and understand its limitations. ‌‌Generalizations from a single or small sample of subjects must be made very carefully (e.g., finding the time a customer is most likely running requires analyzing the run frequencies of many customers). Predicting future values may be impossible if the data hasn’t been prepared well, and even then, there can always be new irregularities in the future. ‌‌Forecasting is usually only stable when you consider the near future. Remember how inaccurate the weather forecast can be when you look it up 10 days in advance. Time-series analysis will never allow you to make exact predictions, only probability distributions of specific values. For example, you can never be sure that a health app user will take more than 10,000 steps on Sunday, only that it is highly likely that they will do it or that you’re 95 % certain they will.‌‌Types of Time-Series AnalysisTime to dive deeper into how time-series analysis can extract information from time-series data. To do this, let’s divide time-series analysis into five distinct types.Exploratory analysisAn exploratory analysis is helpful when you want to describewhatyou see and explainwhyyou see it in a given time series. It essentially entails decomposing the data into trend, seasonality, cyclicity, and irregularities. ‌‌Once the series is decomposed, we can explain what each component represents in the real world and even, perhaps, what caused it. This is not as easy as it may seem and often involves spectral decomposition to find any specific frequencies of recurrences and autocorrelation analysis to see if current values depend on past values.Curve fittingSince time series is a discrete set, you can always tell exactly how many data points it contains.But what if you want to know the value of your time-series parameter at a point in time that is not covered by your data? ‌‌To answer this question, we have to supplement our data with a continuous set—a curve. You can do this in several ways, including interpolation and regression. The former is an exact match for parts of the given time series and is mostly useful for estimating missing data points. On the other hand, the latter is a “best-fit” curve, where you have to make an educated guess about the form of the function to be fitted (e.g., linear) and then vary the parameters until your best-fit criteria are satisfied. ‌‌What constitutes a “best-fit” situation depends on the desired outcome and the particular problem. Using regression analysis, you also obtain the best-fit function parameters that can have real-world meaning, for example, post-run heart rate recovery as an exponential decay fit parameter. In regression, we get a function that describes the best fit to our data even beyond the last record opening the door to extrapolation predictions.ForecastingStatistical inference is the process of generalization from sample to whole. It can be done over time in time-series data, giving way to future predictions or forecasting: from extrapolating regression models to more advanced techniques using stochastic simulations and machine learning. If you want to know more, check out thisarticle about time-series forecasting.Classification and segmentationLet’s start with an example: imagine a family sharing one smartwatch for all of the members' runs. The time-series data generated by the watch will have all the family members’ information mixed up. So how would they distinguish each other's runs? Cue in classification and segmentation, although they’re different types of analysis. ‌‌Classification is looking for patterns in the series and allocating them to a certain number of classes. Segmentation, on the other hand, is separating a time series into a sequence of segments by some desired criteria. ‌‌In our case, this would be the different runs recorded by the watch. Once the series is split (or segmented) into runs, this becomes a classification problem as each runner will have their unique pattern of speed, stride length, etc., distinguishing them from the rest.‌‌As you may have already guessed, problems rarely require just one type of analysis. Still, it is crucial to understand the various types to appreciate each aspect of the problem correctly and formulate a good strategy for addressing it.Visualization and Examples—Run, Overlapping, and Separated ChartsThere are many ways to visualize a time series and certain types of its analysis. A run chart is the most common choice for simple time series with one parameter, essentially just data points connected by lines. ‌‌However, there are usually several parameters you would like to visualize at once. You have two options in this case: overlapping or separated charts. Overlapping charts display multiple series on a single pane, whereas separated charts show individual series in smaller, stacked, and aligned charts, as seen below.Overlapping chart with two y-axes (source)Overlapping chart with two y-axes (source)Separated chart (sea surface temperature anomaly around the Great Barrier Reef) (source)Let’s take a look at three different real-world examples illustrating what we’ve learned so far. To keep things simple and best demonstrate the analysis types, the following examples will be single-parameter series visualized by run charts.Electricity demand in AustraliaStepping away from our health theme, let's explore the time series of Australian monthly electricity demand in the figures below. Visually, it is immediately apparent there is a positive trend, as one would expect with population growth and technological advancement. ‌‌Second, there is a pronounced seasonality to the data, as demand in winter will not be the same as in summer. An autocorrelation analysis can help us understand this better. Fundamentally, this checks the correlation between two points separated by a time delay or lag. ‌‌As we can see in the autocorrelation function (ACF) graph, the highest correlation comes with a delay of exactly 12 months (implying a yearly seasonality) and the lowest with a half-year separation since electricity consumption is highly dependent on the time of year (air-conditioning, daylight hours, etc.). ‌‌Since the underlying data has a trend (it isn’t stationary), as the lag increases, the ACF dies down since the two points are further and further apart, with the positive trend separating them more each year. These conclusions can become increasingly non-trivial when data spans less intuitive variables.‌Monthly electricity demand in Australia and the autocorrelation function (source)Boston Marathon winning timesBack to our health theme from the more exploratory previous example, let’s look at the winning times of the Boston Marathon. The aim here is different: we don’t particularly care why the winning times are such. We want to know whether they have been trending and where we can expect them to go. ‌‌To do this, we need to fit a curve and assess its predictions. But how to know which curve to choose? There is no universal answer to this; however, even visually, you can eliminate a lot of options. In the figure below, we show you four different choices of fitted curves:‌‌‌‌‌‌1. A linear fitf(t) = at + b‌‌2. A piecewise linear fit, which is just several linear fit segments spliced together3. An exponential fitf(t) = aebt+ c‌4. A cubic spline fit that’s like a piecewise linear fit where the segments are cubic polynomials that have to join smoothly‌f(t) = at3+ bt2+ ct + d‌‌Looking at the graph, it’s clear that the linear and exponential options aren’t a good fit. It boils down to the cubic spline and the piecewise linear fits. In fact, both are useful, although for different questions. ‌‌The cubic spline is visually the best historical fit, but in the future (purple section), it trends upward in an intuitively unrealistic way, with the piecewise linear actually producing a far more reasonable prediction. Therefore, one has to be very careful when using good historical fits for prediction, which is why understanding the underlying data is extremely important when choosing forecasting models.Time-series data on the Boston Marathon’s winning times with different fitted curves and their forecasts (source)‌Electrocardiogram analysisAs a final example to illustrate the classification and segmentation types of problems, take a look at the following graph. Imagine wanting to train a machine to recognize certain heart irregularities from electrocardiogram (ECG) readings. ‌‌First, this is a segmentation problem, as you need to split each ECG time series into sequences corresponding to one heartbeat cycle. The dashed red lines in the diagram are the splittings of these cycles. Having done this on both regular and irregular readings, this becomes a classification problem—the algorithm should now analyze other ECG readouts and search for patterns corresponding to either a regular or irregular heartbeat.ECG time series segmented into heartbeat cycles (source)‌Challenges in Handling Time-Series DataAlthough time-series data offers valuable insights, it also presents unique challenges that need to be addressed during analysis.Dealing with missing valuesTime-series data often contains missing or incomplete values, which can adversely affect the accuracy of analysis and modeling. To handle missing values, various techniques likeinterpolationor imputation can be applied, depending on the nature of the data and the extent of missingness.Overcoming noise in time-series dataNoise refers to random fluctuations or irregularities in time-series data, which can obscure the underlying patterns and trends. Filtering techniques, such as moving averages or wavelet transforms, can help reduce noise and extract the essential information from the data.Learn More About Time-Series AnalysisThis was just a glimpse of what time-series analysis offers. By now, you should know that time-series data is ubiquitous. To measure the constant change around you for added efficiency and productivity (whether in life or business),you need to go for it and start analyzing it.‌‌I hope this article has piqued your interest, but nothing compares to trying it out yourself. And for that, you need a robust database to handle the massive time-series datasets.Try Timescale, a modern, cloud-native relational database platform for time series that will give you reliability, fast queries, and the ability to scale infinitely to understand betterwhatis changing,why,andwhen.‌‌Continue your time-series journey:What Is Time-Series Data? (With Examples)What Is Time-Series Forecasting?Time-Series Database: An ExplainerWhat Is a Time-Series Graph With ExamplesWhat Is a Time-Series Plot, and How Can You Create OneGet Started With TimescaleDB With Our TutorialsHow to Write Better Queries for Time-Series Data Analysis With Custom SQL FunctionsSpeeding Up Data Analysis With TimescaleDB and PostgreSQL‌Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/what-is-time-series-analysis-with-examples-and-applications/
2022-01-20T14:34:18.000Z,How to shape sample data with PostgreSQL generate_series() and SQL,"In the lifecycle of any application, developers are often asked to create proof-of-concept features, test newly released functionality, and visualize data analysis. In many cases, the available test data is stale, not representative of normal usage, or simply doesn't exist for the feature being implemented. In situations like this, knowing how to quickly create sample time-series data with native PostgreSQL and SQL functions is a valuable skill to draw upon!In this three-part series on generating sample time-series data, we demonstrate how to use the built-in PostgreSQL function,generate_series(), to more easily create large sets of data to help test various workloads, database features, or just to create fun samples.Inpart 1andpart 2of the series, we reviewed howgenerate_series()works, how to join multiple series using a CROSS JOIN to create large datasets quickly, and finally how to create and use custom PostgreSQL functions as part of the query to generate more realistic values for your dataset. If you haven't usedgenerate_series()much before, we recommend first reading the other two posts.The first one is anintro to the generate_series() function, and the second one showshow to generate more realistic data.With those skills in hand, you can quickly and easily generate tens of millions of rows of realistic-looking data. Even still, there's one more problem that we hinted at in part 2 - all of our data, regardless of how it's formatted or constrained, is still based on therandom()function. This means that over thousands or millions of samples, every device we create data for will likely have the same MAX() and MIN() value, and the distribution of random values over millions of rows for each device generally means that all devices will have similar average values.This third post demonstrates a few methods for influencing how to create data that mimics a desired shape or trend. Do you need to simulate time-series values that cycle over time? What about demonstrating a counter value that resets every so often to test thecounter_agghyperfunction? Are you trying to create new dashboards that display sales data over time, influenced for different months of the year when sales would ebb and flow?Below we'll cover all of these examples to provide you with the final building blocks to create awesome sample data for all of your testing and exploration needs. Remember, however, that these examples are just the beginning. Keep playing. Tweak the formulas or add different relational data to influence the values that get generated so that it meets your use case.Data inceptionTime-series data often has patterns. Weather temperatures and rainfall measurements change in a (mostly) predictable way throughout the year. Vibration measurements from an IoT device connected to an air conditioning system usually increase in the summer and decrease in the winter. Manufacturing data that measures the total units produced per hour (and the percentage of defective units) usually follow a pattern based on shift schedules and seasonal demand.If you want to demonstrate this kind of data without having access to the production dataset, how would you go about it usinggenerate_series()? SQL functions ended up being pretty handy when we discussed different methods for creating realistic-looking data inpart 2. Do you think they might help here? 😉Two options to easily return the row numberRemember, for our purposes we're specifically talking about creating sampletime-series data. Every row increases along the time axis, and if we use the multiplication formula from part 1, we can determine how many rows our sample data query will generate. Using built-in SQL functions, we can quickly start manipulating data values that change with the cycle of time. 💥There are many reasons why it can be helpful to know the ordinal position of each row number in a query result. That's why standard SQL dialects have some variation of therow_number() over()window function. This simple, yet powerful, window function allows us to return the row number of a result set, and can utilize the ORDER BY and PARTITION keywords to further determine the row values.SELECT ts, row_number() over(order by time) AS rownum
FROM generate_series('2022-01-01','2022-01-05',INTERVAL '1 day') ts;

ts                          |rownum|
-----------------------------+------+
2022-01-01 00:00:00.000 -0500|     1|
2022-01-02 00:00:00.000 -0500|     2|
2022-01-03 00:00:00.000 -0500|     3|
2022-01-04 00:00:00.000 -0500|     4|
2022-01-05 00:00:00.000 -0500|     5|In a normal query, this can be useful for tasks like paging data in a web API when there is a need to consistently return values based on a common partition.There's one problem though.row_number() over()requires PostgreSQL (and any other SQL database) to process the query results twice to add the values correctly. Therefore, it's very useful, but also very expensive as datasets grow.Fortunately,PostgreSQL helps us once againfor our specific use case of generating sample time-series data.Through this series of blog posts on generating sample time-series data, we've discussed thatgenerate_series()is aSet Returning Function (SRF). Like the results from a table, set data can be JOINed and queried. Additionally, PostgreSQL provides theWITH ORDINALITYclause that can be applied to any SRF to generate an additional, incrementing BIGINT column. The best part? It doesn't require a second pass through the data in order to generate this value!SELECT ts AS time, rownum
FROM generate_series('2022-01-01','2022-01-05',INTERVAL '1 day') WITH ORDINALITY AS t(ts,rownum);

time                         |rownum|
-----------------------------+------+
2022-01-01 00:00:00.000 -0500|     1|
2022-01-02 00:00:00.000 -0500|     2|
2022-01-03 00:00:00.000 -0500|     3|
2022-01-04 00:00:00.000 -0500|     4|
2022-01-05 00:00:00.000 -0500|     5|Because it serves our purpose and is more efficient, the remainder of this post will useWITH ORDINALITY. However, remember that youcanaccomplish the same results usingrow_number() over()if that's more comfortable for you.Harnessing the row valueWith increasing timestamps and an increasing integer on every row, we can begin to use other functions to create interesting data.Remember from the previous blog posts that calling a function as part of your query executes the function for each row and returns the value. Just like a regular column, however, we don't have to actually emit that column in the final query results. Instead, the function value for that row can be used in calculating values in other columns.As an example, let's modify the previous query. Instead of displaying the row number, let's multiply the value by 2. That is, the function value is treated as an input to a multiplication formula.SELECT ts AS time, 2 * rownum AS rownum_by_two
FROM generate_series('2022-01-01','2022-01-05',INTERVAL '1 day') WITH ORDINALITY AS t(ts,rownum);

time                         |rownum_by_two|
-----------------------------+------+
2022-01-01 00:00:00.000 -0500|     2|
2022-01-02 00:00:00.000 -0500|     4|
2022-01-03 00:00:00.000 -0500|     6|
2022-01-04 00:00:00.000 -0500|     8|
2022-01-05 00:00:00.000 -0500|   10|Easy enough, right? What else can we do with the row number value?Counters with resetMany time-series datasets record values that reset over time, often referred to as counters. The odometer on a car is an example. If you drive far enough, it will ""roll over"" to zero again and start counting upward. The same is true for many utilities, like water and electric meters, that track consumption. Eventually, the total digits will increment to the point where the counter resets and starts from zero again.To simulate this with time-series data, we can use the incrementing row number and after some period of time, reset the count and start over using the modulus operator (%).– This example resets the counter every 10 rows 
WITH counter_rows AS (
	SELECT ts, 
		CASE WHEN rownum % 10 = 0 THEN 10
		     ELSE rownum % 10 END AS row_counter
	FROM generate_series(now() - INTERVAL '5 minutes', now(), INTERVAL '1 second') WITH ORDINALITY AS t(ts, rownum)
)
SELECT ts, row_counter
FROM counter_rows;


ts                         |row_counter|
-----------------------------+-----------+
2022-01-07 13:17:46.427 -0500|          1|
2022-01-07 13:17:47.427 -0500|          2|
2022-01-07 13:17:48.427 -0500|          3|
2022-01-07 13:17:49.427 -0500|          4|
2022-01-07 13:17:50.427 -0500|          5|
2022-01-07 13:17:51.427 -0500|          6|
2022-01-07 13:17:52.427 -0500|          7|
2022-01-07 13:17:53.427 -0500|          8|
2022-01-07 13:17:54.427 -0500|          9|
2022-01-07 13:17:55.427 -0500|         10|
2022-01-07 13:17:56.427 -0500|          1|
… | …By putting the CASE statement inside of the CTE, the counter data can be selected more easily to test other functions. For instance, to see how therate()anddelta()hyperfunctions work, we can usetime_bucket()to group our 1-second readings into 1-minute buckets.WITH counter_rows AS (
	SELECT ts, 
		CASE WHEN rownum % 10 = 0 THEN 10
		     ELSE rownum % 10 END AS row_counter
	FROM generate_series(now() - INTERVAL '5 minutes', now(), INTERVAL '1 second') WITH ORDINALITY AS t(ts, rownum)
)
SELECT time_bucket('1 minute', ts) bucket, 
  delta(counter_agg(ts,row_counter)),
  rate(counter_agg(ts, row_counter))
FROM counter_rows
GROUP BY bucket
ORDER BY bucket;

bucket                       |delta|rate|
-----------------------------+-----+----+
2022-01-07 13:25:00.000 -0500| 33.0| 1.0|
2022-01-07 13:26:00.000 -0500| 59.0| 1.0|
2022-01-07 13:27:00.000 -0500| 59.0| 1.0|
2022-01-07 13:28:00.000 -0500| 59.0| 1.0|
2022-01-07 13:29:00.000 -0500| 59.0| 1.0|
2022-01-07 13:30:00.000 -0500| 26.0| 1.0|time_bucket()outputs the starting time of the bucket, which based on our date math forgenerate_series()produces four complete buckets of 1-minute aggregated data, and two partial buckets - one for the minute we are currently in, and a second bucket for the partial 5 minutes ago. We can see that the delta correctly calculates the difference between the last and first readings of each bucket, and the rate of change (the increment between each reading) correctly displays a unit of one.What are some other ways we can use these PostgreSQL functions to generate different shapes of data to help you explore other features of SQL and TimescaleDB quickly?Increasing trend over timeWith the knowledge of how to create an ordinal value for each row of data produced bygenerate_series(), we can explore other ways of generating useful time-series data. Because the row number value will always increase, we can easily produce a random dataset that always increases over time but has some variability to it. Consider this a very rough representation of daily website traffic over the span of two years.SELECT ts, (10 + 10 * random()) * rownum as value FROM generate_series
       ( '2020-01-01'::date
       , '2021-12-31'::date
       , INTERVAL '1 day') WITH ORDINALITY AS t(ts, rownum);Sample daily website traffic growing over time with random daily valuesIn reality this chart isn't very realistic or representative. Any website that gains and loses viewers upwards of 50% per day probably isn't going to have great long-term success. Don't worry, we can do better with this example after we learn about another method for creating shaped data using sine waves.Simple cycles (sine wave)Using the built-insin()andcos()PostgreSQL functions, we can generate data useful for graphing and testing functions that need a predictable data trend. This is particularly useful for testing TimescaleDB downsampling hyperfunctions likelttborasap. These functions can take tens of thousands (or millions) of data points and return a smaller, but still accurately representative dataset for graphing.We'll start with a basic example that produces one row per day, for 30 days. For each row number value, we'll get the sine value that can be used to graph a wave.–- subtract 1 from the row number for wave to start
-- at zero radians and produce a more representative chart
SELECT  ts,
 cos(rownum-1) as value
FROM generate_series('2021-01-01','2021-01-30',INTERVAL '1 day') WITH ORDINALITY AS t(ts, rownum);

ts                         |value                 |
-----------------------------+--------------------+
2021-01-01 00:00:00.000 -0500|                 1.0|
2021-01-02 00:00:00.000 -0500|  0.5403023058681398|
2021-01-03 00:00:00.000 -0500| -0.4161468365471424|
2021-01-04 00:00:00.000 -0500| -0.9899924966004454|
2021-01-05 00:00:00.000 -0500| -0.6536436208636119|
2021-01-06 00:00:00.000 -0500| 0.28366218546322625|
… | …Unfortunately, the graph of this SINE wave doesn't look all that appealing. For one month of daily data points, we only have ~6 distinct data points from peak to peak of each wave.Cosine wave graph using daily values for one monthThe reason our sine wave is so jagged is that sine and cosine values are measured in radians (based on 𝞹), not degrees. A complete cycle (peak-to-peak) on a sine wave happens from zero to 2*𝞹 (~6.28…). Therefore, every ~6 rows of data will produce a complete period in the wave - unless we find a way to modify that value.To take control over the sine/cosine values, we need to think about how to modify the data based on the date range and interval (how many rows) and what we want the wave to look like.This means we need to take a quick trip back to math class to talk about radians.Math class flashback!Step back with me for a minute to primary school and your favorite math subject - Algebra 2 (or Trigonometry as the case may be). How many hours did you spend working with graph paper (or graphing calculators) determining the amplitude, period, and shift of a sine or cosine graph?Sine wave period and amplitudeIf you reach even further into your memory, you might remember this formula which allows you to modify the various aspects of a wave.Mathematical formula for modifying the shape and values of a sine waveThere's a lot here, I know. Let's primarily focus on the two numbers that matter most for our current use case:X = the ""number of radians"", which is the row number in our datasetB = a value to multiply the row number by, to decrease the ""radian"" value for each row(A, C, and D change the height and placement of the wave, but to start, we want to elongate each period and provide more ""points"" on the line to graph.)Let's start with a small dataset example, generating cosine data for three months of daily timestamps with no modifications.SELECT  ts, 
cos(rownum) as value
FROM generate_series('2021-01-01','2021-03-31',INTERVAL '1 day') WITH ORDINALITY AS t(ts, rownum);


ts                         |value                  |
-----------------------------+---------------------+
2021-01-01 00:00:00.000 -0500|   0.5403023058681398|
2021-01-02 00:00:00.000 -0500|  -0.4161468365471424|
2021-01-03 00:00:00.000 -0500|  -0.9899924966004454|
2021-01-04 00:00:00.000 -0500|  -0.6536436208636119|
2021-01-05 00:00:00.000 -0500|  0.28366218546322625|
2021-01-06 00:00:00.000 -0500|    0.960170286650366|
2021-01-07 00:00:00.000 -0500|   0.7539022543433046|
2021-01-08 00:00:00.000 -0500| -0.14550003380861354|
2021-01-09 00:00:00.000 -0500|  -0.9111302618846769|
… | …
2021-03-29 00:00:00.000 -0400|   0.9993732836951247|
2021-03-30 00:00:00.000 -0400|   0.5101770449416689|
2021-03-31 00:00:00.000 -0400|  -0.4480736161291701|Cosine wave with daily data for three monthsIn this example, we see ~14 peaks in our wave because there are 90 points of data and without modification, the wave will have a period (peak-to-peak) every ~6.28 points. To lengthen the cycle, we need to perform some simple division.[cycle modifying value] = 6.28/[total interval (rows) per cycle]Using the same 3 months of generated daily values, let's see how to modify the data to lengthen the period of the wave.One cycle per month (30 days)If we want our daily data to cycle every 30 days, multiply our row number value by 6.28/30.6.28/30 = .209 (the row number radians modifier)SELECT  ts, cos(rownum * 6.28/30) as value
FROM generate_series('2021-01-01','2021-03-31',INTERVAL '1 day') WITH ORDINALITY AS t(ts, rownum);Cosine wave using daily data for three months, adjusted to have monthly periodsOne cycle per quarter (90 days)6.28/90 = .07 (this is our radians modifier)SELECT  ts, cos(rownum * 6.28/90) as value
FROM generate_series('2021-01-01','2021-03-31',INTERVAL '1 day') WITH ORDINALITY AS t(ts, rownum);Cosine wave using daily data for three months, adjusted to have one 3-month periodTo modify the overall length of the period, you need to modify the row number value based on the total number of rows in the result and the granularity of the timestamp.Here are some example values that you can use to modify the wave period based on the interval used withgenerate_series().generate_series() intervalDesired period lengthDivide 6.28 by…daily1 month30daily3 months90hourly1 day24hourly1 week168hourly1 month720minute1 hour60minute1 day1440Modifying the wave amplitude and shiftAnother tweak we can make to our wave data is to change the amplitude (difference between the min and max peaks) and, as necessary, shift the wave up or down on the Y-axis.To do this, multiply the cosine value by the value that maximum value you want the wave to have. For example, we can multiply the monthly cycle data by 10, which changes the overall minimum and maximum values of the data.SELECT  ts, 10 * cos(rownum * 6.28/30) as value
FROM generate_series('2021-01-01','2021-03-31',INTERVAL '1 day') WITH ORDINALITY AS t(ts, rownum);Cosine wave using daily data for three months with increased amplitudeNotice that the min/max values are now from -10 to 10.We can take it one step further by adding a value to the output which will shift the final values up or down on the Y-axis. In this example, we modified the previous query by adding 10 to the value of each row which results in values from 0 to 20.SELECT  ts, 10 + 10 * cos(rownum * 6.28/30) as value
FROM generate_series('2021-01-01','2021-03-31',INTERVAL '1 day') WITH ORDINALITY AS t(ts, rownum);Cosine wave using daily data for three months with shifted min/max values on the Y-axisWhy spend so much time showing you how to generate and manipulate sine or cosine wave data, especially when we rarely see repeatable data this smooth in real life?One of the main advantages of using consistent, predictable data like this in testing is that you can easily tell if your application, charting tools, and query are working as expected. Once you begin adding in unpredictable, real-life data, it can be difficult to determine if the data, query, or application are producing unexpected results. Quickly generating known data with a specific pattern can help rule out errors with the query, at least.The second advantage of using a known dataset is that it can be used to shape and influence the results of other queries. Earlier in this post, we demonstrated a very simplistic example of increasing website traffic by multiplying the row number and a random value. Let's look at how we can join both datasets to create a better shape for the sample website traffic data.Better website traffic samplesOne of the key takeaways from this series of posts is thatgenerate_series()returns a set of data that can be JOINed and manipulated like data from a regular table. Therefore, we can join together our rough ""website traffic"" data and our sine wave to produce a smoother, more realistic set of data to experiment with. SQL for the win!Overall this is one of the more complex examples we've presented, utilizing multiple common table expressions (CTE) to break the various sets into separate tables that we can query and join. However, this also means that you can independently modify the time range and other values to change the data that is generated from this query for your own experimentation.-- This is the generate series data
-- with a ""short"" date to join with later
WITH daily_series AS ( 
	SELECT ts, date(ts) AS day, rownum FROM generate_series
       ( '2020-01-01'
       , '2021-12-31'
       , '1 day'::interval) WITH ORDINALITY AS t(ts, rownum)
),
-- This selects the time, ""day"", and a 
-- random value that represents our daily website visits
daily_value AS ( 
	SELECT ts, day, rownum, random() AS val
    FROM daily_series
    ORDER BY day
),
-- This cosine wave dataset has the same ""day"" values which allow 
-- it to be joined to the daily_value easily. The wave value is used to modify
-- the ""website"" value by some percentage to smooth it out 
-- in the shape of the wave.
daily_wave AS ( 
	SELECT
       day,
       -- 6.28 radians divided by 180 days (rows) to get 
       -- one peak every 6 months (twice a year)
       1 + .2 * cos(rownum * 6.28/180) as p_mod
       FROM daily_series
       day
)
-- (500 + 20 * val) = 500-520 visits per day before modification
-- p_mod = an adjusted cosine value that raises or lowers our data each day
-- row_number = a big incremental value for each row to quickly increase ""visits"" each day
SELECT dv.ts, (500 + 20 * val) * p_mod * rownum as value
FROM daily_value dv
	INNER JOIN daily_wave dw ON dv.DAY=dw.DAY
    order by ts;Combining wave data and random increasing data to shape the data patternWithout much effort, we are able to generate a time-series dataset, use two different SQL functions, and join multiple sets together to create fun, graphical data. In this example, our traffic peaks twice a year (every ~180 days) during July and late December.But we don't have to stop there. We can carry our website traffic example one step further by applying just a little more control over how much the data increases or decreases during certain periods.Once again, relational data to the rescue!Influence the pattern with relational dataAs a final example, let's consider one other type of data that we can include in our queries that influence the final generated values - relational data. Although we've been using data that was created usinggenerate_series()to produce some fun and interesting sample datasets, we can just as easily JOIN to other data in our database to further manipulate the final result.There are many ways you could JOIN to and use additional data depending on your use case and the type of time-series data you're trying to mimic. For example:IoT data from weather sensors:store the typical weekly temperature highs/lows in a database table and use those values as input to therandom_between()function we created in post 2Stock data analysis:store the dates for quarterly disclosures and a hypothetical factor that will influence the impact on stock price moving forwardSales or website traffic:store the monthly or weekly change observed in a typical sales cycle. Does traffic or sales increase a quarter-end? What about during the end-of-year holiday season?To demonstrate this, we'll use the fictitious website traffic data from earlier in this post. Specifically, we've decided that we want to see a spike in traffic during June and December.First, we create a regular PostgreSQL table to store the numerical month (1-12) and a float value which will be used to modify our generated data (up or down). This will allow us to tweak the overall shape for a given month.CREATE TABLE overrides (
	m_val INT NOT NULL,
	p_inc FLOAT4 NOT null
);

INSERT INTO overrides(m_val, p_inc) VALUES 
	(1,.1.04), – 4% residual increase from December
	(2,1),
	(3,1),
	(4,1),
	(5,1),
	(6,1.10),-- June increase of 10%
	(7,1),
	(8,1),
	(9,1),
	(10,1),
	(11,1.08), -- 8% early shoppers sales/traffic growth
	(12,1.18); -- 18% holiday increaseUsing this simple dataset, let's first join it to the ""simplistic"" query that had randomly growing data over time.WITH daily_series AS (
-- a random value that increases over time based on the row number
SELECT ts, date_part('month',ts) AS m_val, (10 + 10*random()) * rownum as value FROM generate_series
       ( '2020-01-01'::date
       , '2021-12-31'::date
       , INTERVAL '1 day') WITH ORDINALITY AS t(ts, rownum)
)
-- join to the `overrides` table to get the 'p_inc' value 
-- for the month of the current row
SELECT ts, value * p_inc AS value FROM daily_series ds
INNER JOIN overrides o ON ds.m_val=o.m_val
ORDER BY ts;Sample website traffic for two years, modified during some months with relational dataJoining to theoverridestable based on the month of each data point, we are able to multiply the percentage increase (p_inc) value and the fake website traffic value to influence the trend of our data during specific time periods.Combining everything we've learned and taking this example one step further, we can enhance the cosine data query with the same monthly override values to tweak our fake, cyclical time-series data that represents growing website traffic with a more realistic shape.​​-- This is the generate series data
-- with a ""short"" date to join with later
WITH daily_series AS ( 
	SELECT ts, date(ts) AS day, rownum FROM generate_series
       ( '2020-01-01'
       , '2021-12-31'
       , '1 day'::interval) WITH ORDINALITY AS t(ts, rownum)
),
-- This selects the time, ""day"", and a 
-- random value that represents our daily website visits
-- 'm_val' will be used to join with the 'overrides' table
daily_value AS ( 
	SELECT ts, day, date_part('month',ts) as m_val, rownum, random() AS val
    FROM daily_series
    ORDER BY day
),
-- This cosine wave dataset has the same ""day"" values which allow 
-- it to be joined to the daily_value easily. The wave value is used to modify
-- the ""website"" value by some percentage to smooth it out 
-- in the shape of the wave.
daily_wave AS ( 
	SELECT
       day,
       -- 6.28 radians divided by 180 days (rows) to get 
       -- one peak every 6 months (twice a year)
       1 + .2 * cos(rownum * 6.28/180) as p_mod
       FROM daily_series
       day
)
-- (500 + 20 * val) = 500-520 visits per day before modification
-- p_mod = an adjusted cosine value that raises or lowers our data each day
-- row_number = a big incremental value for each row to quickly increase ""visits"" each day
-- p_inc = a monthly adjustment value taken from the 'overrides' table
SELECT dv.ts, (500 + 20 * val) * p_mod * rownum * p_inc as value
FROM daily_value dv
	INNER JOIN daily_wave dw ON dv.DAY=dw.DAY
    inner join overrides o on dv.m_val=o.m_val
    order by ts;Sample website traffic for two years combined with sine wave data and modified during some months with relational dataWrapping it upIn this 3rd and final blog post of our series about generating sample time-series datasets, we demonstrated how to add shape and trend into your sample time-series data (e.g., increasing web traffic over time and quarterly sales cycles) using built-in SQL functions and relational data. With a little bit of math mixed in, we learned how to manipulate the pattern of generated data, which is particularly useful for visualizing time-series data and learning analytical PostgreSQL or TimescaleDB functions.To see some of these examples in action, watch my video on creating realistic sample data:If you have questions about usinggenerate_series()or have any questions about TimescaleDB, pleasejoin our community Slack channel, where you'll find an active community and a handful of the Timescale team most days.If you want to try creating larger sets of sample time-series data using generate_series() and see how the exciting features of TimescaleDB work,sign up for a free 30-day trialorinstall and manage it on your instances. (You can also learn more byfollowing one of our many tutorials.)Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-shape-sample-data-with-postgresql-generate_series-and-sql/
2021-09-23T13:01:37.000Z,AWS Lambda for Beginners: Overcoming the Most Common Challenges,"Learn how to solve three common problems that you’ll likely encounter when using AWS Lambda: adding dependencies to your function, overcoming the 250MB function size limitation for application code and dependencies, and building a continuous deployment pipeline.Developing data pipelines - an automated process that acquires, transforms, and stores data - is a common need for most application developers in today’s always-on, 24/7 economy. Serverless architectures provide the perfect solution for managing these pipelines in a lightweight nature, with elastic pricing and almost no operational overhead.Whether you arerecording IoT metrics, buildingstockorcrypto applications, or poweringmachine learning models, serverless infrastructure is the go-to solution to move time-series data around. (If you’re new to time-series data and want to learn more, check out our “what is time-series data?” blog post.)In the world of serverless service providers, AWS Lambda isone of the most popular choices. This article will go over three common challenges you’ll likely face while building data pipelines with AWS Lambda and a step-by-step solution to each.You will see how to add external dependencies to your AWS Lambda function, overcome the 250MB package limit, and set up continuous deployment.What is AWS Lambda?Before we get into the nitty-gritty of solving challenges, let’s first understand what AWS Lambdais.AWS Lambda, or just Lambda, is a serverless computing service that enables you to run code without provisioning or managing servers. Lambda makes it easy to call or automatically trigger Java, Go, PowerShell, Node.js, C#, Python, and Ruby code. You can also connect AWS Lambda withAWS API Gatewayto expose your function as an API endpoint or automatically run the function periodically usingAWS EventBridgeorAWS SNS/SQS.Let’s cover some of the issues and considerations for Lambda!Size matters!When it comes to Lambda, you can be sure of one thing: size absolutely matters! There arehard limitsfor a Lambda package size which can be a deal-breaker depending on your use case. For example, suppose your Python function depends onPandas,NLTK, and a database connector library. In this scenario, the whole unzipped package size is already over the limit of 250MB, so you won’t be able touploadyour function, let alone run it.Historically, Lambda has been an excellent choice only for small-to-medium-sized functions with small dependencies. The moment you needed to include a larger dependency, it became questionable whether Lambda was the best tool for your process. Although you could have used another serverless option likeAWS Fargateor bootstrapped a specializedAWS EC2instance to process your data, they generally required more operational experience and overhead.Fortunately, during theAWS re:Invent 2020 event, Amazon introduced a new feature that made it possible to deploy larger functions within Lambda and use more dependencies than before:container support.That’s right! You can now upload Docker images as Lambda functions with an upper limit of 10GB! This feature is a massive upgrade from the previous 250MB hard limit: you containerize your function code, upload it to ECR, and then deploy it directly to Lambda.Pros and cons of AWS Lambda for data pipelinesBefore deploying your data pipeline in the cloud, it’s important to look at your options and decide which serverless provider and tool/service to use that fits your needs.Here’s a non-exhaustive list of major pros and cons of Lambda:AWS Lambda prosCheaper than regular hosting (like EC2)Data pipelines usually get triggered by an event or are scheduled to run at specific time intervals — they don’t need to be up and running all the time. With Lambda, you only pay for computing costs that your function actually uses (in milliseconds) — no minimum execution time.No server managementLambda is also sometimes referred to as a FaaS (function as a service) wherein the only thing that the user needs to develop and manage is the function itself. No provisioning or infrastructure management.Seamless scalabilityYou are not required to do anything manually to address scalability because it’s automatically managed by Lambda to accommodate the rate of incoming invocations. Does your function suddenly need more resources because there’s more data to process? No problem, Lambda can handle scaling up and down.AWS Lambda cons50MB zipped and 250MB unzipped limit deployment package limitLambda functions are supposed to be small and only do one specific task. But still, if your function needs to include multiple dependencies, it’s easy to go over the limits.Max. 15 minute execution timeYou can set the maximum timeout for the function, but it cannot run longer than 15 minutes. This limitation makes Lambda unsuitable for long-running processes.Cumbersome deploymentWhether you opt to create a ZIP archive or a container image, deploying your function is not easy, especially with dependencies. Every time you want to deploy a new version of your function, you need to package your files and dependencies again to upload it. Hence, continuous deployment automation is essential to effectively manage your functions, saving you a lot of time and effort.Of these three common issues, the package size limitation and managing continuous deployments are often the most troublesome. For the remainder of this article, we’d like to explore workarounds to help mitigate these specific issues.Using TimescaleDB with AWS LambdaIf you are not yet familiar withTimescaleDB, it is a time-series database packaged as a PostgreSQL extension. It has features and optimizations to ingest, store and analyze time-series data efficiently. TimescaleDB works well with tools and services that support PostgreSQL, including Lambda.TimescaleDB users often use AWS Lambda to load time-series data into their tables. Furthermore, Lambda can be used for transforming, fetching, and performing other data operations on TimescaleDB tables. Let’s cover some examples and sample Lambda function code to give you inspiration!Example use cases for AWS Lambda with TimescaleDB:Insert data from a third-party APIInsert data from another databaseFetch data from TimescaleDBCreate a data API for TimescaleDBPerform ETL operationsSince TimescaleDB is a PostgreSQL extension, you can access your TimescaleDB instance using your programming language’s database connector library (likeJDBCin Java orPsycopg2in Python).Here’s a Python example using Psycopg2 that shows how to connect to aTimescaleDB hypertableand query it inside a Lambda function:import json
import psycopg2
import psycopg2.extras
import os
 
def lambda_handler(event, context):
   db_name = os.environ['DB_NAME']
   db_user = os.environ['DB_USER']
   db_host = os.environ['DB_HOST']
   db_port = os.environ['DB_PORT']
   db_pass = os.environ['DB_PASS']
   conn = psycopg2.connect(user=db_user, database=db_name, host=db_host,
                           password=db_pass, port=db_port)
  
   sql = ""SELECT * FROM hypertable WHERE time > WHERE time > NOW() - INTERVAL '2 weeks'
   cursor = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
   cursor.execute(sql)
   result = cursor.fetchall()
   return {
       'statusCode': 200,
       'body': json.dumps(result, default=str),
       'headers': {
           ""Content-Type"": ""application/json""
       }
   }In this Lambda function code snippet, after creating a connection object, we query the hypertable to return all records from the past two weeks.At the end of the function, we return this result set in JSON format.For a simple example like this, a straightforward Lambda function can be coded, uploaded, and triggered without much hassle. However, because TimescaleDB is a time-series database that often deals with tens of thousands of rows (or more) in a single transaction, we find that users often struggle with the problems we discussed earlier.Let’s explore how to overcome these obstacles for larger data pipelines using AWS Lambda!Problem #1: how to add external dependenciesSuppose you want to interact with TimescaleDB (or any other external database) in your function to build a data pipeline or a data API. In that case, you need at least one external dependency: a database connector library. With Lambda, you can useLambda Layersto include the dependencies of the function.A Lambda Layer is an archive containing additional code, such as libraries. Additionally, you can use these libraries across multiple functions, which can come in handy if you build data pipelines because each function will have to use the connector library. Instead of adding common libraries to each function ZIP package, you can refer to the layers within the function which reduces complexity and eases the management of dependency versions across all of your functions!Upload function dependencies using Lambda LayersSince we are using Python, let’s see how you can upload the Psycopg2 library as a Lambda Layer! It’s a little tricky as you need to use the compiled version of the library.1) Download and unzip the compiled Psycopg2 library:wget https://github.com/jkehler/awslambda-psycopg2/archive/refs/heads/master.zip
unzip master.zip2) In the directory you downloaded the library to, copy the Psycopg2 files into a new directory called /python/psycopg2/:cd awslambda-psycopg2-master/
mkdir -p python/psycopg2/
cp -r psycopg2-3.8/* python/psycopg2/3) Zip the python directory and upload the zipped file as a Lambda layer:zip -r psycopg2_layer.zip python/
aws lambda publish-layer-version --layer-name psycopg2 \
--description ""psycopg2 for Python3.8"" --zip-file fileb://psycopg2_layer.zip \
--compatible-runtimes python3.84) In the AWS Lambda console, check to see if your psycopg2 has been uploaded as a Lambda layer:Psycopg2 library uploaded as a Lambda LayerAnd there you have it, Psycopg2 uploaded as a Layer! You can now build out your data pipelines in Lambda that connect to any PostgreSQL database, like TimescaleDB. 🎉If you need a more detailed tutorial on working with Lambda Layers, read ourCreate a data API for TimescaleDB tutorialto learn how to use Lambda Layers and API Gateway to create a time-series data API.With Lambda Layers, you can add and share dependencies between your functions.But, it’s important to note that the final package size includes your function code and any layers that you associate with the function. This can be crucial if your function is close to going over the 250MB limit.Let’s see how to overcome this limit next!Problem #2: how to overcome the 250MB package limitWhat if your function’s dependencies include something bigger than a simple connector library? Or suppose you have multiple shared dependencies and using them together in a function puts you over the 250MB limit. Fear not, there’s an elegant solution to this problem:use a Docker container as your Lambda function.As I mentioned, AWS announcedcontainer support for Lambda, which allows the package size to be up to 10GB. This gives you much more flexibility regarding what external libraries and dependencies you use in your Lambda function. This process includes containerizing your function and all its dependencies with Docker, then uploading it to AWS ECR. Finally, connect this image on ECR to Lambda.What is AWS ECR (Elastic Container Registry)?AWS ECR is a Docker container registry system where you can store, manage, share, and deploy your Docker containers. ECR works well with other AWS services, including Lambda. It allows you to deploy your containers directly from this registry to Lambda.Containerize your function, upload it to AWS ECR, and connect it to Lambda.Let’s look at a quick example of how to use a Docker image with Lambda in Python:1) Create your Lambda function2) Add a requirements file containing your dependencies# requirements.txt:
requests
pandas
...3) Create the Dockerfile, which is based on an AWS-provided Lambda imageFROM public.ecr.aws/lambda/python:3.8
COPY requirements.txt .RUN pip install -r requirements.txt
CMD [""function.handler""]4) Build the image and upload it to ECR5) Create the Lambda function using the imageAfter completing these steps, you have set up a Docker container as your Lambda function. For a detailed step-by-step tutorial on using Docker containers with Lambda, head over to ourPull and ingest data from a third-party APIdocs to learn how to use Docker to build a 3rd party API ingestion function with Lambda and TimescaleDB.Being able to exceed the 250MB limit gives you more freedom and enables you to use Lambda for more use cases than before. For example, if you couldn’t import your favorite data processing library before because it was too big for Lambda, with containers, you can freely do it.As with most things, however, the ability to run a larger, more complex Lambda function also brings a new layer of operational overhead - managing and deploying a Docker image aside from your function code.To make that process easier, let’s look at how we can automate the management of the Docker image and continuously deploy changes to AWS Lambda!Problem #3: how to set up continuous deploymentCI/CD is part of every software development process. In the Lambda ecosystem, there are multiple ways to build a CI/CD pipeline. One popular choice isGitHub Actions. Whether you’re using a serverless framework likeSAMor zipping your dependencies then uploading it to Lambda, automating this process can save a lot of time packaging and deploying functions.Push your function to GitHub and create a GitHub Actions workflow file to create the CD pipelineWith GitHub Actions, you manage the process through a YAML file in your function source repository, which instructs GitHub how to package and deploy your image any time something happens, like merging a pull request into the `main` branch.An overview of the process for setting up continuous deployment between a GitHub repository and Lambda, using GitHub Actions generally looks like this:1) Create a new function in the AWS console2) Push your function and its dependencies to a new GitHub repository3) Add your AWS credentials to the repository using GitHub secrets4) Set up GitHub Actions to auto-deploy when there are new changes on the master branchYAML file:name: deploy to lambda
on:
  # Trigger the workflow on push requests on the master branch
  push:
    branches:
      - master
jobs:

  deploy_source:
    name: deploy lambda from source
    runs-on: ubuntu-latest
    steps:
      - name: checkout source code
        uses: actions/checkout@v1
      - name: default deploy
        uses: appleboy/lambda-action@master
        with:
          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws_region: ${{ secrets.AWS_REGION }}
          function_name: lambda-cd
          source: function.pyIn this overview example, GitHub will deploy changes to AWS Lambda any time you push a commit to the `main` branch, using the credentials you specified as part of the setup. Depending on your use case, the deployment strategy and process could be more complex.For details on building a continuous deployment pipeline between GitHub and Lambda, check out ourLambda continuous deployment with GitHub Actionstutorial.Wrapping upYou’ve learned about AWS Lambda and how to solve three common challenges that you’ll likely run into as you use it to build data pipelines. We’ve provided a sample process to add external dependencies to your function, using Lambda Layers – and you’ve seen how to workaround Lambda’s 250MB package size limit (Docker containers to the rescue!). And finally, you know how to use GitHub Actions to set up continuous deployment with Lambda.If you want to read more detailed Lambda tutorials, go to theTimescaleDB with AWS Lambdadocumentation section to dig deeper. If you don’t already have TimescaleDB installed, it’s easiest to sign up for afree Timescaleaccount, or you can alwaysinstall it on your own machine. You can also getfree access to AWS Lambda.Have questions? Join 7000+ TimescaleDB users and engineers in theTimescaleDB community Slackto learn more about working with time-series data!Finally, here are some inspirational tutorials to get you started with building time-series data pipelines :)Analyze cryptocurrency market dataAnalyze historical intraday stock dataAnalyze data using TimescaleDB continuous aggregates and hyperfunctionsGetting started with Grafana and TimescaleDBIngest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/aws-lambda-for-beginners-overcoming-the-most-common-challenges/
2021-11-11T14:51:33.000Z,Generating More Realistic Sample Time-Series Data With PostgreSQL generate_series(),"In this three-part series on generating sample time-series data, we demonstrate how to use the built-in PostgreSQL function,generate_series(), to more easily create large sets of data to help test various workloads, database features, or just to create fun samples.In part 1 of the series, we reviewed howgenerate_series()works, including the ability to join multiple series into a larger table of time-series data - through a feature known as a CROSS (or Cartesian) JOIN. We ended the first post by showing you how to quickly calculate the number of rows a query will produce and modify the parameters forgenerate_series()to fine-tune the size and shape of the data.However, there was one problem with the data we could produce at the end of the first post. The data that we were able to generate was very basic and not very realistic. Without more effort, using functions likerandom()to generate values doesn't provide much control over precisely what numbers are produced, so the data still feels more fake than we might want.This second post will demonstrate a few ways to create more realistic-looking data beyond a column or two of random decimal values. Read on for more.Inpart 3of this blog series adds one final tool to the mix - combining the data formatting techniques below with additional equations and relational data to shape your sample time-series output into something that more closely resembles real-life applications.By the end of this series, you'll be ready to test almost any feature that TimescaleDB offers and create quick datasets for your testing and demos!A brief review of generate_series()In the first post, we demonstrated howgenerate_series()(a Set Returning Function) could quickly create a data set based on a range of numeric values or dates. The generated data is essentially an in-memory table that can quickly create large sets of sample data.-- create a series of values, 1 through 5, incrementing by 1
SELECT * FROM generate_series(1,5);

generate_series|
---------------|
              1|
              2|
              3|
              4|
              5|


-- generate a series of timestamps, incrementing by 1 hour
SELECT * from generate_series('2021-01-01','2021-01-02', INTERVAL '1 hour');

    generate_series     
------------------------
 2021-01-01 00:00:00+00
 2021-01-01 01:00:00+00
 2021-01-01 02:00:00+00
 2021-01-01 03:00:00+00
 2021-01-01 04:00:00+00
...We then discussed how the data quickly becomes more complex as we join the various sets together (along with some value returning functions) to create a multiple of both sets together.This example from the first post joined a timestamp set, a numeric set, and therandom()function to create fake CPU data for four fake devices over time.-- there is an implicit CROSS JOIN between the two generate_series() sets
SELECT time, device_id, random()*100 as cpu_usage 
FROM generate_series('2021-01-01 00:00:00','2021-01-01 04:00:00',INTERVAL '1 hour') as time, 
generate_series(1,4) device_id;


time               |device_id|cpu_usage          |
-------------------+---------+-------------------+
2021-01-01 00:00:00|        1|0.35415126479989567|
2021-01-01 01:00:00|        1| 14.013393572770028|
2021-01-01 02:00:00|        1|   88.5015939122006|
2021-01-01 03:00:00|        1|  97.49037810105996|
2021-01-01 04:00:00|        1|  50.22781125586846|
2021-01-01 00:00:00|        2|  46.41196423062297|
2021-01-01 01:00:00|        2|  74.39903569177027|
2021-01-01 02:00:00|        2|  85.44087332221935|
2021-01-01 03:00:00|        2|  4.329394730750735|
2021-01-01 04:00:00|        2| 54.645873866589056|
2021-01-01 00:00:00|        3|  63.01888063314749|
2021-01-01 01:00:00|        3|  21.70606884856987|
2021-01-01 02:00:00|        3|  32.47610779097485|
2021-01-01 03:00:00|        3| 47.565982341726354|
2021-01-01 04:00:00|        3|  64.34867263419619|
2021-01-01 00:00:00|        4|   78.1768041898232|
2021-01-01 01:00:00|        4|  84.51505102850199|
2021-01-01 02:00:00|        4| 24.029611792753514|
2021-01-01 03:00:00|        4|  17.08996115345549|
2021-01-01 04:00:00|        4| 29.642690955760997|And finally, we talked about how to calculate the total number of rows your query would generate based on the time range, the interval between timestamps, and the number of ""things"" for which you are creating fake data.Range of readingsLength of intervalNumber of ""devices""Total rows1 year1 hour435,0401 year10 minutes1005,256,0006 months5 minutes1,00052,560,000Still,the main problem remains.Even if we can generate 50 million rows of data with a few lines of SQL, the data we generate isn't very realistic. It's all random numbers, with lots of decimals and minimal variation.As we saw in the query above (generating fake CPU data), any columns of data that we add to the SELECT query are added to each row of the resulting set. If we add static text (like 'Hello, Timescale!'), that text is repeated for every row. Likewise, adding a function as a column value will be called one time for each row of the final set.That's what happened with therandom()function in the CPU data example. Every row has a different value because the function is called separately for each row of generated data. We can use this to our advantage to begin making the data look more realistic.With a little more thought and custom PostgreSQL functions, we can start to bring our sample data ""to life.""What is realistic data?This feels like a good time to make sure we're on the same page. What do I mean by ""realistic"" data?Using the basic techniques we've already discussed allows you to create a lot of data quickly. In most cases, however, you often know what the data you're trying to explore looks like. It's probably not a bunch of decimal or integer values. Even if the data you're trying to mimicarejust numeric values, they likely have valid ranges and maybe a predictable frequency.Take our simple example of CPU and temperature data from above. With just two fields, we have a few choices to make if we want the generated data tofeelmore realistic.Is CPU a percentage? Out of 100% or are we representing multi-core CPUs that can present as 200%, 400%, or 800%?Is temperature measured in Fahrenheit or Celsius? What are reasonable values for CPU temperature in each unit? Do we store temperature with decimals or as an integer in the schema?What if we added a ""note"" field to the schema for messages that our monitoring software might add to the readings from time to time? Would every reading have a note or just when a threshold was reached? Is there a special diagnostic message at the top of each hour that we need to replicate in some way?Usingrandom()and static text by themselves allows us to generatelotsof data with many columns, but it's not going to be very interesting or as useful in testing features in the database.That's the goal of the second and third posts in this series, helping you to produce sample data that looks more like the real thing without much extra work. Yes, itwillstill be random, but it will be random within constraints that help you feel more connected to the data as you explore various aspects of time-series data.And, by using functions, all of the work is easily reusable from table to table.Walk before you runIn each of the examples below, we'll approach our solutions much as we learned in elementary math class:show your work! It's often difficult to create a function or procedure in PostgreSQL without playing with a plain SQL statement first. This abstracts away the need to think about function inputs and outputs at the outset so that we can focus on how the SQL works to produce the value we want.Therefore, the examples below show you how to get a value (random numbers, text, JSON, etc.) in a SELECT statement first before converting the SQL into a function that can be reused. This kind of iterative process is a great way to learn features of PostgreSQL, particularly when it's combined withgenerate_series().So, take one foot and put it in front of the other, and let's start creating better sample data.Creating more realistic numbersIn time-series data, numeric values are often the most common data type. Using a function likerandom()without any other formatting creates very… well... random (and precise) numbers with lots of decimal points. While itworks,the values aren'trealistic. Most users and devices aren't tracking CPU usage to 12+ decimals. We need a way to manipulate and constrain the final value that's returned in the query.For numeric values, PostgreSQL provides many built-in functions to modify the output. In many cases, usinground()andfloor()with basic arithmetic can quickly start shaping the data in a way that better fits your schema and use case.Let's modify the example query for getting device metrics, returning values for CPU and temperature. We want to update the query to ensure that the data values are ""customized"" for each column, returning values within a specific range and precision. Therefore, we need to apply a standard formula to each numeric value in our SELECT query.Final value = random() * (max allowed value - min allowed value) + min allowed valueThis equation will always generate a decimal value between (and inclusive of) the min and max value. Ifrandom()returns a value of 1, the final output will equal the maximum value. Ifrandom()returns a value of 0, then the result will equal the minimum value. Any other number thatrandom()returns will produce some output between the min and max values.Depending on whether we want a decimal or integer value, we can further format the ""final value"" of our formula withround()andfloor().This example produces a reading every minute for one hour for 10 devices. The cpu value will always fall between 3 and 100 (with four decimals of precision), and the temperature will always be an integer between 28 and 83.SELECT
  time,
  device_id,
  round((random()* (100-3) + 3)::NUMERIC, 4) AS cpu,
  floor(random()* (83-28) + 28)::INTEGER AS tempc
FROM 
	generate_series(now() - interval '1 hour', now(), interval '1 minute') AS time, 
	generate_series(1,10,1) AS device_id;


time                         |device_id|cpu    |tempc        |
-----------------------------+---------+-------+-------------+
2021-11-03 12:47:01.181 -0400|        1|53.7301|           61|
2021-11-03 12:48:01.181 -0400|        1|34.7655|           46|
2021-11-03 12:49:01.181 -0400|        1|78.6849|           44|
2021-11-03 12:50:01.181 -0400|        1|95.5484|           64|
2021-11-03 12:51:01.181 -0400|        1|86.3073|           82|
…|...|...|...By using our simple formula and formatting the result correctly, the query produced the ""curated"" output (random as it is) we wanted.The power of functionsBut there's also a bit of a letdown here, isn't there? Typing that formula repeatedly for each value - trying to remember the order of parameters and when I need to cast a value - will become tedious quickly. After all, you only have so manykeystrokes left.The solution is to create and use PostgreSQL functions that can take the inputs we need, do the correct calculations, and return the formatted value that we want. There aremanyways we could accomplish a calculation like this in a function. Use this example as a starting place for your learning and exploration.Note:In this example, I chose to return the value from this function as anumericdata type because it can return values thatlooklike integers (no decimals) or floats (decimals). As long as the return values are inserted into a table with the intended schema, this is a ""trick"" to visually see what we expect - an integer or a float. In general, thenumericdata type will often perform worse in queries and features like compression because of hownumericvalues are represented internally. We recommend avoidingnumerictypes in schema design whenever possible, preferring the float or integer types instead./*
 * Function to create a random numeric value between two numbers
 * 
 * NOTICE: We are using the type of 'numeric' in this function in order
 * to visually return values that look like integers (no decimals) and 
 * floats (with decimals). However, if inserted into a table, the assumption
 * is that the appropriate column type is used. The `numeric` type is often
 * not the correct or most efficient type for storing numbers in a table.
 */
CREATE OR REPLACE FUNCTION random_between(min_val numeric, max_val numeric, round_to int=0) 
   RETURNS numeric AS
$$
 DECLARE
 	value NUMERIC = random()* (min_val - max_val) + max_val;
BEGIN
   IF round_to = 0 THEN 
	 RETURN floor(value);
   ELSE 
   	 RETURN round(value,round_to);
   END IF;
END
$$ language 'plpgsql';This example function uses the minimum and maximum values provided, applies the ""range"" formula we discussed earlier, and finally returns anumericvalue that either has decimals (to the specified number of digits) or not. Using this function in our query, we can simplify creating formatted values for sample data, and it cleans up the SQL, making it easier to read and use.SELECT
  time,
  device_id,
  random_between(3,100, 4) AS cpu,
  random_between(28,83) AS temperature_c
FROM 
	generate_series(now() - interval '1 hour', now(), interval '1 minute') AS time, 
	generate_series(1,10,1) AS device_id;This query provides the same formatted output, but now it's much easier to repeat the process.Creating more realistic textWhat about text? So far, in both articles, we've only discussed how to generate numeric data. We all know, however, that time-series data often contain more than just numeric values. Let's turn to another common data type: text.Time-series data often contains text values. When your schema contains log messages, item names, or other identifying information stored as text, we want to generate sample text that feels more realistic, even if it's random.Let's consider the query used earlier that creates CPU and temperature data for a set of devices. If the devices were real, the data they create might contain an intermittent status message of varying length.To figure out how to generate this random text, we will follow the same process as before, working directly in a stand-alone SQL query before moving our solution into a reusable function. After some initial attempts (and ample Googling), I came up with this example for producing random text of variable length using a defined character set. As with therandom_between()function above, this can be modified to suit your needs. For instance, it would be fairly easy to get unique, random hexadecimal values by limiting the set of characters and lengths.Let your creativity guide you.WITH symbols(characters) as (VALUES ('ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 0123456789 {}')),
w1 AS (
	SELECT string_agg(substr(characters, (random() * length(characters) + 1) :: INTEGER, 1), '') r_text, 'g1' AS idx
	FROM symbols,
generate_series(1,10) as word(chr_idx) -- word length
	GROUP BY idx)
SELECT
  time,
  device_id,
  random_between(3,100, 4) AS cpu,
  random_between(28,83) AS temperature_c,
  w1.r_text AS note
FROM w1, generate_series(now() - interval '1 hour', now(), interval '1 minute') AS time, 
	generate_series(1,10,1) AS device_id
ORDER BY 1,2;

time                         |device_id|cpu     |temperature_c|note      |
-----------------------------+---------+--------+-------------+----------+
2021-11-03 16:49:24.218 -0400|        1| 88.3525|           50|I}3U}FIsX9|
2021-11-03 16:49:24.218 -0400|        2| 29.5313|           53|I}3U}FIsX9|
2021-11-03 16:49:24.218 -0400|        3| 97.6065|           70|I}3U}FIsX9|
2021-11-03 16:49:24.218 -0400|        4| 96.2170|           40|I}3U}FIsX9|
2021-11-03 16:49:24.218 -0400|        5| 53.2318|           82|I}3U}FIsX9|
2021-11-03 16:49:24.218 -0400|        6| 73.7244|           56|I}3U}FIsX9|In this case, it was easier to generate a random value inside of a CTE that we could reference later in the query. However, this approach has one problem that's pretty easy to spot in the first few rows of returned data.While the CTE does create random text of 10 characters (go ahead and run it a few times to verify), the value of the CTE is generated once each time and then cached, repeating the same result over and over for every row. Once we transfer the query into a function, we expect to see a different value for each row.For this second example function to generate ""words"" of random lengths (or no text at all in some cases), the user will need to provide an integer for the minimum and maximum length of the generated text. After some testing, we also added a simple randomizing feature.Notice the IF...THEN condition that we added. Any time the generated number is divided by five and has a remainder of zero or one, the function will not return a text value. There is nothing special about this approach to providing randomness to the frequency of the output, so feel free to adjust this part of the function to suit your needs./*
 * Function to create random text, of varying length
 */
CREATE OR REPLACE FUNCTION random_text(min_val INT=0, max_val INT=50) 
   RETURNS text AS
$$
DECLARE 
	word_length NUMERIC  = floor(random() * (max_val-min_val) + min_val)::INTEGER;
	random_word TEXT = '';
BEGIN
	-- only if the word length we get has a remainder after being divided by 5. This gives
	-- some randomness to when words are produced or not. Adjust for your tastes.
	IF(word_length % 5) > 1 THEN
	SELECT * INTO random_word FROM (
		WITH symbols(characters) AS (VALUES ('ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz 0123456789 '))
		SELECT string_agg(substr(characters, (random() * length(characters) + 1) :: INTEGER, 1), ''), 'g1' AS idx
		FROM symbols
		JOIN generate_series(1,word_length) AS word(chr_idx) on 1 = 1 -- word length
		group by idx) a;
	END IF;
	RETURN random_word;
END
$$ LANGUAGE 'plpgsql';When we use this function to add random text to our sample time-series query, notice that the text is random in length (between 2 and 10 characters) and frequency.SELECT
  time,
  device_id,
  random_between(3,100, 4) AS cpu,
  random_between(28,83) AS temperature_c,
  random_text(2,10) AS note
FROM generate_series(now() - interval '1 hour', now(), interval '1 minute') AS time, 
	generate_series(1,10,1) AS device_id
ORDER BY 1,2;

time                         |device_id|cpu     |temperature_c|note     |
-----------------------------+---------+--------+-------------+---------+
2021-11-04 14:17:03.410 -0400|        1| 86.5780|           67|         |
2021-11-04 14:17:03.410 -0400|        2|  3.5370|           76|pCVBp AZ |
2021-11-04 14:17:03.410 -0400|        3| 59.7085|           28|kMrr     |
2021-11-04 14:17:03.410 -0400|        4| 69.6153|           46|3UdA     |
2021-11-04 14:17:03.410 -0400|        5| 33.0906|           56|d0sSUilx |
2021-11-04 14:17:03.410 -0400|        6| 44.2837|           74|         |
2021-11-04 14:17:03.410 -0400|        7| 14.2550|           81|TOgbHOU  |Hopefully, you're starting to see a pattern. Usinggenerate_series()and some custom functions can help you create time-series data of many shapes and sizes.We've demonstrated ways to create more realistic numbers and text data because they are the primary data types used in time-series data. Are there any other data types included with time-series data that you might need to generate with your sample data?What about JSON values?Creating sample JSONNote:The sample queries below create JSON strings as the output with the intention that it would be inserted into a table for further testing and learning. In PostgreSQL, JSON string data can be stored in a JSON or JSONB column, each providing different features for querying and displaying the JSON data. In most circumstances, JSONB is the preferred column type because it provides more efficient storage and the ability to create indexes over the contents. The main downside is that the actual formatting of the JSON string, including the order of the keys and values, is not retained and may be difficult to reproduce exactly. To better understand the differences of when you would store JSON string data with one column type over the other, pleaserefer to the PostgreSQL documentation.PostgreSQL has supported JSON and JSONB data types for many years. With each major release, the feature set for working with JSON and overall query performance improves. In a growing number of data models, particularly when REST or Graph APIs are involved, storing extra meta information as a JSON document can be beneficial. The data is available if needed while facilitating efficient queries on serialized data stored in regular columns.We used a design pattern similar to this in ourNFT Starter Kit. The OpenSea JSON API used as the data source for the starter kit includes many properties and values for each asset and collection. A lot of the values weren't helpful for the specific analysis in that tutorial. However, we knew that some of the values in the JSON properties could be useful in future analysis, tutorials, or demonstrations. Therefore, we stored additional metadata about assets and collections in a JSONB field to query it if needed. Still, it didn't complicate the schema design for otherwise common data likenameandasset_id.Storing data in a JSON field is also a common practice in areas like IIoT device data. Engineers usually have an agreed-upon schema to store and query metrics produced by the device, followed by a ""free form"" JSON column that allows engineers to send error or diagnostic data that changes over time as hardware is modified or updated.There are several approaches to add JSON data to our sample query. One added challenge is that JSON data includes both a key and a value, along with the possibility of numerous levels of child object nesting. The approach you take will depend on how complex you want the PostgreSQL function to be and the end goal of the sample data. In this example, we'll create a function that takes an array of keys for the JSON and generates random numerical values for each key without nesting. Generating the JSON string in SQL from our values is straightforward, thanks to built-in PostgreSQL functions for reading and writing JSON strings. 🎉As with the other examples in this post, we'll start by using a CTE to generate a random JSON document in a stand-alone SELECT query to verify that the result is what we want. Remember, we'll observe the same issue we had earlier when generating random text in the stand-alone query because we are using a CTE. The JSON is random every time the query runs, but the string is reused for all rows in the result set. CTE's are materialized once for each reference in a query, whereas functions are called again for every row. Because of this, we won't observe random values in each row until we move the SQL into a function to reuse later.WITH random_json AS (
SELECT json_object_agg(key, random_between(1,10)) as json_data
    FROM unnest(array['a', 'b']) as u(key))
  SELECT json_data, generate_series(1,5) FROM random_json;

json_data       |generate_series|
----------------+---------------+
{""a"": 6, ""b"": 2}|              1|
{""a"": 6, ""b"": 2}|              2|
{""a"": 6, ""b"": 2}|              3|
{""a"": 6, ""b"": 2}|              4|
{""a"": 6, ""b"": 2}|              5|We can see that the JSON data is created using our keys (['a','b']) with numbers between 1 and 10. We just have to create a function that will create random JSON data each time it is called. This function will always return a JSON document with numeric integer values for each key we provide for demonstration purposes. Feel free to enhance this function to return more complex documents with various data types if that's a requirement for you.CREATE OR REPLACE FUNCTION random_json(keys TEXT[]='{""a"",""b"",""c""}',min_val NUMERIC = 0, max_val NUMERIC = 10) 
   RETURNS JSON AS
$$
DECLARE 
	random_val NUMERIC  = floor(random() * (max_val-min_val) + min_val)::INTEGER;
	random_json JSON = NULL;
BEGIN
	-- again, this adds some randomness into the results. Remove or modify if this
	-- isn't useful for your situation
	if(random_val % 5) > 1 then
		SELECT * INTO random_json FROM (
			SELECT json_object_agg(key, random_between(min_val,max_val)) as json_data
	    		FROM unnest(keys) as u(key)
		) json_val;
	END IF;
	RETURN random_json;
END
$$ LANGUAGE 'plpgsql';With therandom_json()function in place, we can test it in a few ways. First, we'll simply call the function directly without any parameters, which will return a JSON document with the default keys provided in the function definition (""a"", ""b"", ""c"") and values from 0 to 10 (the default minimum and maximum value).SELECT random_json();

random_json             |
------------------------+
{""a"": 7, ""b"": 3, ""c"": 8}|Next, we'll join this to a small numeric set fromgenerate_series().SELECT device_id, random_json() FROM generate_series(1,5) device_id;

device_id|random_json              |
---------+-------------------------+
        1|{""a"": 2, ""b"": 2, ""c"": 2} |
        2|                         |
        3|{""a"": 10, ""b"": 7, ""c"": 1}|
        4|                         |
        5|{""a"": 7, ""b"": 1, ""c"": 0} |Notice two things with this example.First, the data is different for each row, showing that the function gets called for each row and produces different numeric values each time. Second, because we kept the same random output mechanism from therandom_text()example, not every row includes JSON.Finally, let's add this into the sample query for generating device data that we've used throughout this article to see how to provide an array of keys (""building"" and ""rack"") for the generated JSON data.SELECT
  time,
  device_id,
  random_between(3,100, 4) AS cpu,
  random_between(28,83) AS temperature_c,
  random_text(2,10) AS note,
  random_json(ARRAY['building','rack'],1,20) device_location
FROM generate_series(now() - interval '1 hour', now(), interval '1 minute') AS time, 
	generate_series(1,10,1) AS device_id
ORDER BY 1,2;


time                         |device_id|cpu     |temperature_c|note     |device_location             |
-----------------------------+---------+--------+-------------+---------+----------------------------+
2021-11-04 16:19:22.991 -0400|        1| 14.7614|           70|CTcX8 2s4|                            |
2021-11-04 16:19:22.991 -0400|        2| 62.2618|           81|x1V      |{""rack"": 4, ""building"": 5}  |
2021-11-04 16:19:22.991 -0400|        3| 10.1214|           50|1PNb     |                            |
2021-11-04 16:19:22.991 -0400|        4| 96.3742|           29|aZpikXGe |{""rack"": 12, ""building"": 4} |
2021-11-04 16:19:22.991 -0400|        5| 22.5327|           30|lM       |{""rack"": 2, ""building"": 3}  |
2021-11-04 16:19:22.991 -0400|        6| 57.9773|           44|         |{""rack"": 16, ""building"": 5} |
...There are just so many possibilities for creating sample data withgenerate_series(), PostgreSQL functions, and some custom logic.Putting it all togetherLet's put what we've learned into practice, using these three functions to create and insert ~1 million rows of data and then query it with the hyperfunctionstime_bucket(),time_bucket_ng(),approx_percentile()andtime_weight(). To do this, we'll create two tables: one will be a list of computer hosts and the second will be a hypertable that stores fake time-series data about the computers.Step 1: Create the schema and hypertableCREATE TABLE host (
	id int PRIMARY KEY,
	host_name TEXT,
	LOCATION jsonb
);

CREATE TABLE host_data (
	date timestamptz NOT NULL,
	host_id int NOT NULL,
	cpu double PRECISION,
	tempc int,
	status TEXT	
);

SELECT create_hypertable('host_data','date');Step 2: Generate and insert data-- Insert data to create fake hosts
INSERT INTO host
SELECT id, 'host_' || id::TEXT AS name, 
	random_json(ARRAY['building','rack'],1,20) AS LOCATION
FROM generate_series(1,100) AS id;


-- insert ~1.3 million records for the last 3 months
INSERT INTO host_data
SELECT date, host_id,
	random_between(5,100,3) AS cpu,
	random_between(28,90) AS tempc,
	random_text(20,75) AS status
FROM generate_series(now() - INTERVAL '3 months',now(), INTERVAL '10 minutes') AS date,
generate_series(1,100) AS host_id;Step 3: Query data usingtime_bucket()andtime_bucket_ng()-- Using time_bucket(), query the average CPU and max tempc
SELECT time_bucket('7 days', date) AS bucket, host_name,
	avg(cpu),
	max(tempc)
FROM host_data
JOIN host ON host_data.host_id = host.id
WHERE date > now() - INTERVAL '1 month'
GROUP BY 1,2
ORDER BY 1 DESC, 2;


-- try the experimental time_bucket_ng() to query data in month buckets
SELECT timescaledb_experimental.time_bucket_ng('1 month', date) AS bucket, host_name,
	avg(cpu) avg_cpu,
	max(tempc) max_temp
FROM host_data
JOIN host ON host_data.host_id = host.id
WHERE date > now() - INTERVAL '3 month'
GROUP BY 1,2
ORDER BY 1 DESC, 2;Step 4: Query data using toolkit hyperfunctions-- query all host in building 10 for 7 day buckets
-- also try the new percentile approximation function to 
-- get the p75 of data for each 7 day period
SELECT time_bucket('7 days', date) AS bucket, host_name,
	avg(cpu),
	approx_percentile(0.75,percentile_agg(cpu)) p75,
	max(tempc)
FROM host_data
JOIN host ON host_data.host_id = host.id
WHERE date > now() - INTERVAL '1 month'
	AND LOCATION -> 'building' = '10'
GROUP BY 1, 2
ORDER BY 1 DESC, 2;



-- To test time-weighted averages, we need to simulate missing
-- some data points in our host_data table. To do this, we'll
-- randomly select ~10% of the rows, and then delete them from the
-- host_data table.
WITH random_delete AS (SELECT date, host_id FROM host_data
	 JOIN host ON host_id = id WHERE 
	date > now() - INTERVAL '2 weeks'
	ORDER BY random() LIMIT 20000
)
DELETE FROM host_data hd
USING random_delete rd
WHERE hd.date = rd.date
AND hd.host_id = rd.host_id;


-- Select the daily time-weighted average and regular average
-- of each host for building 10 for the last two weeks.
-- Notice the variation in the two numbers because of the missing data.
SELECT time_bucket('1 day',date) AS bucket,
	host_name,
	average(time_weight('LOCF',date,cpu)) weighted_avg,
	avg(cpu) 
FROM host_data
	JOIN host ON host_data.host_id = host.id
WHERE LOCATION -> 'building' = '10'
AND date > now() - INTERVAL '2 weeks'
GROUP BY 1,2
ORDER BY 1 DESC, 2;In a few lines of SQL, we created 1.3 million rows of data and were able to test four different functions in TimescaleDB, all without relying on any external source. 💪Still, you may notice one last issue with the values in ourhost_datatable (even though the values are not more realistic in nature). By usingrandom()as the basis for our queries, the calculated numeric values all tend to have an equal distribution within the specified range which causes the average of the values to always be near the median. This makes sense statistically, but it highlights one other area of improvement to the data we generate. In the third post of this series, we'll demonstrate a few ways to influence the generated values to provide shape to the data (and even some outliers if we need them).Reviewing our progressWhen using a database like TimescaleDB or testing features in PostgreSQL, generating a representative dataset is a beneficial tool to have in your SQL toolbelt.In the first post, we learned how to generate lots of data by combining the result sets of multiplegenerate_series()functions. Using the implicitCROSS JOIN, the total number of rows in the final output is a product of each set together. When one of the data sets contains timestamps, the output can be used to create time-series data for testing and querying.The problem with our initial examples was that the actual values we generated were random and lacked control over their precision - and all of the data was numeric. So in this second post, we demonstrated how to format the numeric data for a given column and generate random data of other types, like text and JSON documents. We also added an example in the text and JSON functions that created randomness in how often the values were emitted for each of those columns.Again, all of these are building block examples for you to use, creating functions that generate the kind of data you need to test.To see some of these examples in action, watch my video on creating realistic sample data:Inpart 3of this series, we will demonstrate how to add shape and trends into your sample time-series data (e.g., increasing web traffic over time and quarterly sales cycles) using the formatting functions in this post in conjunction with relational lookup tables and additional mathematical functions. Knowing how to manipulate the pattern of generated data is particularly useful for visualizing time-series data and learning analytical PostgreSQL or TimescaleDB functions.If you have questions about usinggenerate_series()or have any questions about TimescaleDB, pleasejoin our community Slack channel, where you'll find an active community and a handful of the Timescale team most days.If you want to try creating larger sets of sample time-series data usinggenerate_series()and see how the exciting features of TimescaleDB work,sign up for a free 30-day trialorinstall and manage it on your instances. (You can also learn more byfollowing one of our many tutorials.)Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/generating-more-realistic-sample-time-series-data-with-postgresql-generate_series/
2022-12-13T14:00:34.000Z,10 Facts About Time-Series Data You Should Know,"You can use it on your smartphone to know whether it’s going to rain tomorrow, on smart home devices that will play music for you (hey, Alexa!), and some countries are even making the most of itto build smart cities and speed up vehicle control.Time-series data is everywhereand helps you track changes over time, allowing you to understand what, why, and when something happened.Time-series analysis takes time-series data a step further, enabling you to extract new insights andforecast changes or trends.As more and more applications are built on it, the need to manage, store, and analyze that timestamped information swiftly and efficiently is increasing for developers and data scientists worldwide.If you are new to the time-series world, let’s start by explaining 10 facts about time-series data so you can better grasp what you’re working with:1. It has time stamps.2. It is append-only.3. There is usually a lot of it.4. It can have a life cycle.5. It is relentless.6. It is everywhere.7. It needs to be available in real time.8. It allows you to forecast.9. It loves to be graphed.10. It can be difficult to host.Watch the video for a more in-depth explanation of each of these points. And if you needthe best database to handle time-series dataor are especially struggling with number 10, we've got you covered:try Timescale for free for 30 days(no credit card required) orself-host using TimescaleDB.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/10-facts-about-time-series-data-you-should-know/
2022-09-08T12:27:00.000Z,What Is Time-Series Forecasting?,"Astime-series databecomes ubiquitous, measuring change is vital to understanding the world. Here at Timescale, we use our PostgreSQL and TimescaleDB superpowers to generate insights into the data to seewhatandhow things changed, but alsowhenthey changed—that’s the beauty of time-series data. But, if you have data showing past and present trends, can you predict the future? Cue in time-series forecasting.In the simplest terms,time-series forecastingis a technique that utilizes historical and current data to predict future values over a period of time or a specific point in the future.By analyzing data that we stored in the past, we can make informed decisions that can guide our business strategy and help us understand future trends.Some of you may be asking yourselves what the difference is between time-series forecasting and algorithmic predictions using, for example, machine learning. Well, machine learning techniques such asrandom forest,gradient boosting regressor, andtime delay neural networkscan be used to extrapolate time-series data, but they are far from the only available options or the best ones (as you will see in this article). The most important property of a time-series algorithm is the ability to extrapolate patterns outside of the domain of training data, which most machine learning techniques cannot do by default. This is where specialized time-series forecasting techniques come in.There are plenty of forecasting techniques to choose from, and this article will help you acquire a basic understanding of the most popular ones. From simple linear regression models to complex and vast neural networks, each forecasting method has its own benefits and drawbacks.Let’s check them out.The value of Bitcoin (BTC) as time-series data (source)Applications of Time-Series ForecastingQuite a few industries and scientific fields are utilizing time-series forecasting. Some of the most relevant ones include:Business planningControl engineeringCryptocurrencytrendsFinancial marketsModeling disease spreadingPattern recognitionResources allocationSignal processingSports analyticsStatisticsWeather forecastingAs you can see, the list is already quite long, but the truth is that anyone who has access to accurate historical data can utilize time-series analysis methods to forecast future developments and trends.When Is Time-Series Forecasting Useful?Even though time-series forecasting may seem like a universally applicable technique, there are some limitations that developers need to be aware of. Because forecasting isn’t a strictly defined method but rather a combination of data analysis techniques, analysts, and data scientists need to consider the limitations that the prediction models hold, as well as the data itself.The most crucial step when considering time-series forecasting isunderstanding your datamodel and knowingwhich business questionsneed to be answered using this data. By diving into the problem domain, a developer can more easily distinguish random fluctuations from stable and constant trends in historical data. This will come in handy when tuning the prediction model to generate the best forecasts and even considering which forecasting method to use.When using time-series analysis, some data limitations need to be considered. Common problems include generalizing from a single data source and difficulty in obtaining appropriate measurements and accurately identifying the correct model to represent the data.What to Consider When You Do Time-Series ForecastingThere are quite a few factors associated with time-series forecasting, but the most important ones include the following:Amount of dataData qualitySeasonalityTrendsUnexpected eventsTheamount of datais probably the most important factor (assuming that the data is accurate). A good rule of thumb would bethe more data we have, the better our model will generate forecasts. This also makes it much easier for our model to distinguish between trends and noise in the data.✨Editor's Note:Optimizing your database ingest rate is critical when working with large amounts of data—learn how to do it in this blog post.Data qualityentails some basic requirements, such as having no duplicates, a standardized data format, and for the data to be collected consistently or at regular intervals.Seasonalitymeans that there are distinct periods of time when the data contains consistent irregularities. For example, if an online web shop analyzed its sales history, it would be evident that the holiday season results in an increased amount of sales. In this example, we can deduce the correlation intuitively, but there are many other examples where analysis methods such as time-series forecasting are needed to detect such consumer behavior.Trendsare probably the most important information you are looking for. They indicate whether a variable in the time series will increase or decrease in a given period. We can also calculate the probability of a trend in order to make even more informed decisions with our data.Unexpected events(sometimes also referred to as noise or irregularities) can always occur, and we need to consider that when creating a prediction model. They present noise in historical data, and they are also not predictable.Overview of Time-Series Forecasting MethodsBelow you can find a basic overview of several forecasting methods that we covered and the theory behind them:Time-series decompositionTime-series regression modelsExponential smoothingARIMA modelsNeural networksTBATSTime-Series DecompositionTime-series decomposition is a method for explicitly modeling the data as a combination ofseasonal,trend, cycle,andremaindercomponents instead of modeling it with temporal dependencies and autocorrelations. It can either be performed as a standalone method for time-series forecasting or as the first step in better understanding your data.When using a decomposition model, you need to forecast future values for each of the components above and then add these predictions together to find the most accurate overall forecast. Some of the most relevant forecasting techniques using decomposition areSeasonal-Trend decomposition using LOESS,Bayesian structural time-series(BSTS), andFacebook Prophet.Time-series decomposition refers to a technique that decomposes time-series data into the following four components:TrendCycleSeasonalityRemainderDecomposition of a used car sales data set (source)Decomposition based on rates of changeDecomposition based on rates of change is a crucial technique when it comes to analyzing seasonal adjustments. The technique constructs several component series, which, when combined (using additions and multiplications), result in the original time series. Each of the components has a certain characteristic or type of behavior, and they usually include:Tt: The trend component at time t describes the long-term progression of the time series. A trend is present when there is a consistent increase or decrease in the direction of the data. The trend component isn’t constrained to a linear function.Ct: The cyclical component at timetreflects repeated but non-periodic fluctuations. The duration of these fluctuations depends on the nature of the time series.St: The seasonal component at timetreflects seasonality (seasonal variation). Such a seasonal pattern can be found in time series that are influenced by seasonal factors. Seasonality usually occurs in a fixed and known period (for example, holiday seasons).It: The irregular component (or ""noise"") at timetrepresents random and irregular influences. It can also be considered the remainder of the time series after other components have been removed.Additive decompositionAdditive decomposition implies that time-series data is a function of the sum of its components. This can be represented with the following equation:yt= Tt+ Ct+ St+ Itwhere ytis the time-series data, Ttis the trend component, Ctis the cycle component, Stis the seasonal component, and Itis the remainder.Multiplicative decompositionInstead of using addition to combine the components, multiplicative decomposition defines time-series data as a function of the product of its components. In the form of an equation:yt= Tt* Ct* St* ItThe question is how to identify a time series as additive or multiplicative. The answer is in its variation. If the magnitude of the seasonal component is dynamic and changes over time, it’s safe to assume that the series is multiplicative. If the seasonal component is constant, the series is additive.Some methods combine the trend and cycle components into one trend-cycle component. It can be referred to as the trend component even when it contains visible cycle properties. For example, when usingseasonal-trend decomposition with LOESS, the time series is decomposed into seasonal, trend, and irregular (also called noise) components, where the cycle component is included in the trend component.Time-Series Regression ModelsTime-series regression is a statistical method of forecasting future values based on historical data. The forecast variable is also called the regressand, dependent or explained variable. The predictor variables are sometimes called the regressors, independent or explanatory variables. Regression algorithms attempt to calculate the line of best fit for a given dataset. For example, a linear regression algorithm could try to minimize the sum of the squares of the differences between the observed value and predicted value to find the best fit.Let’s look at one of the simplest regression models, simple linear regression. The regression model describes a linear relationship between the forecast variable y and a simple predictor variable x:yt= β0+ β1* xt+ εtThe coefficients β0and β1denote the line's intercept and slope. The slope β1represents the average predicted change in y resulting from a one unit increase in x:Simple linear regression model example (source)It’s important to note that the observations aren’t perfectly aligned on the straight line but are somewhat scattered around it. Each of the observations ytis made up of a systematic component of the model (β0+ β1* xt) and an “error” component (εt). The error component doesn’t have to be an actual error; the term encompasses any deviations from the straight-line model.As you can see, a linear model is very limited in approximating underlying functions, which is why other regression models may be more useful, likeLeast squares estimationandNonlinear regression.Exponential SmoothingWhen it comes to time-series forecasting,data smoothingcan tremendously improve the accuracy of our predictions by removing outliers from a time-series dataset. This leads to increased visibility of distinct and repeating patterns that would otherwise be hidden between the noise.Exponential smoothing is a rule-of-thumb technique for smoothing time-series data using the exponential window function. Whereas the simple moving average method weighs historical data equally to make predictions about the future, exponential smoothing uses exponential functions to calculate decreasing weights over time. Different types of exponential smoothing includesimple exponential smoothingandtriple exponential smoothing(also known as the Holt-Winters method).Stationary time-series smoothing using different smoothing rates (EWMA) (source)ARIMA ModelsAutoRegressive Integrated Moving Average, or ARIMA, is a forecasting method that combines both an autoregressive model and a moving average model.Autoregressionuses observations from previous time steps to predict future values using a regression equation. An autoregressive model utilizes a linear combination of past variable values to make forecasts:Thus, an autoregressive model of order p can be written as:yt= c + ϕ1yt-1+ ϕ2yt−2+ ⋯ + ϕpyt−p+ εtwhere εtis white noise. This is like a multiple regression model but with delayed values of ytas predictors. We refer to this as an AR(p) model, an autoregressive model of order p.On the other hand, amoving average modeluses a linear combination of forecast errors for its predictions:yt= c + εt+ θ1εt−1+ θ2εt−2+ ⋯ + θqεt−qwhere εtrepresents white noise. We refer to this as an MA(q) model, a moving average model of order q. The value of εt is not observed, which means we can’t classify it as a regression in the usual sense.If we combine differencing with autoregression and a moving average model, we obtain a non-seasonal ARIMA model. The full model can be represented with the following equation:y′t= c + ϕ1y′t−1+ ⋯ + ϕpy′t−p+ θ1εt−1+ ⋯ + θqεt−q+ εtwhere y′tis the differenced series (more on differencing can be foundhere). The “predictors” on the right-hand side are a combination of the lagged values ytand lagged errors. This model is called an ARIMA( p, d, q) model. The parameters of the model are:p: the order of the autoregressive componentd: the degree of first differencing involvedq: the order of the moving average partThe SARIMA model (Seasonal ARIMA) is an extension of the ARIMA model. This is achieved by adding a linear combination of seasonal past values and forecast errors.Predicting taxicab pickups in Times Square with TimescaleDB (source)Neural NetworksNeural networks are also gaining traction regarding tasks such as classification and prediction. Recent studies have shown that a neural network is able to approximate any continuous function sufficiently well for time-series forecasting. While classical methods like ARMA and ARIMA assume a linear relationship between inputs and outputs, neural networks are not bound by this constraint. They are able to approximate any nonlinear function without prior knowledge about the properties of the data series.Neural networks such as multilayer perceptrons (MLPs) offer multiple advantages that make them worth considering:Robust to noise: Neural networks are not only robust to noise when it comes to input data but also robust in the mapping function. This can come in handy when working with data that contains missing values.Nonlinear support: Neural networks are not bound to strong assumptions and a rigid mapping function. They are able to learn from new linear and nonlinear relationships continuously.Multivariate inputs: Multivariate forecasting is supported because the number of input features is completely variable.Multi-step forecasts: The number of output values is variable as well.TBATSA lot of time series contain complex and multiple seasonal patterns (e.g., hourly data containing a daily pattern, weekly pattern, and annual pattern). The most popular models (e.g., ARIMA and exponential smoothing) can only account for one seasonality.A TBATS model can deal with complex seasonalities (e.g., non-integer seasonality, non-nested seasonality, and large-period seasonality) with no seasonality constraints, making it possible to create detailed, long-term forecasts. But there is also a drawback to using TBATS models. They can be slow when calculating predictions, especially with long time series.TBATS is an acronym for some of the most important features that the model offers:T: Trigonometric seasonalityB: Box-Cox transformationA: ARIMA errorsT: TrendS: Seasonal componentsConclusionTime-series forecasting is a powerful method for predicting future trends and values in time-series data. Time-series forecasting may hold tremendous value for your business development if you have access to historical information with a time component. While there is a myriad of forecasting methods to choose from, most of them are focused on specific situations and types of data, which makes it relatively easy to choose the right one.If you are interested in time-series forecasting, take a look atthis tutorialabout analyzing Cryptocurrency market data.By using a time-series databaselike TimescaleDB, you can ditch complex analysis techniques that require a lot of custom code and instead use the SQL query language to generate insights.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/what-is-time-series-forecasting/
2022-01-13T17:19:35.000Z,How to Store and Analyze NFT Data in a Relational Database,"In this post, we share behind-the-scenes technical insights about how we designed and built theTimescale NFT Starter Kit. We discuss the technical thought process behind how we selected a reliable source of NFT data, how we designed a database schema to ingest and analyze NFT sales data, and how we handled some of the challenges behind data ingestion and analysis in our quest to explore the NFT world from a data-driven perspective.We recently launched the TimescaleNFT Starter Kit, a step-by-step guide to get up and running with collecting, storing, and analyzing NFT data. Our goal was to help developers who are crypto-enthusiasts (or just crypto-curious) bring data to their NFT purchasing decisions, build more complex NFT tracking projects or just learn about the space from a more data-driven perspective (see ourannouncement blogfor more details).The Timescale NFT Starter Kit consists of:A database schema to efficiently store and query data.SQL queries to use as a starting point for your own analysis.Pre-built dashboards and templates in popular data visualization tools likeApache SupersetandGrafanafor visualizing your data analysis.A Python script to collect real-time data from OpenSea, as well as a sample dataset to play around with.A tutorial that takes you step-by-step.The project received a lot oflove on Twitterand the 20Time Travel Tiger NFTs, which we created to reward the developers who completed the NFT Analysis tutorial first, were all claimed within 14 days of us publishing the NFT Starter Kit. Even though all the Time Travel Tiger NFTs were claimed, theNFT Analysis Tutorialis still available for you to complete and is a great starting point for getting started with NFT data and unearthing insights from it (it’s a fun weekend project!).In addition to the tutorial, we createdpre-built dashboardswhich enable you to start exploring NFT trends in less than 5 minutes using popular visualization tools Apache Superset and Grafana. See ourdemo videosfor a walk-through of installing and using the pre-built dashboards.One of the two pre-built Apache Superset dashboards in the NFT Starter Kit.One of the two pre-built Grafana dashboards in the NFT Starter KitGiven the growth of the NFT space, with companies likeOpenSeareaching billion dollar valuations and the launch of new marketplaces likeLooksRare, there is growing developer interest in collecting and analyzing NFT data. To help fellow developers who are interested in NFT data, we want to share the technical thought process behind how we designed and built the Timescale NFT Starter Kit.In the rest of this post, we’ll discuss the core decisions behind the following aspects of the Timescale NFT Starter Kit:Selecting a data sourceSelecting data to analyzeDesigning a database schemaFormulating SQL queries for analysisWe’re openly sharing our experience working with NFT data in the hopes that it benefits other developers who are building projects around collecting, storing, and analyzing NFT data. The NFT space is relatively new and so we hope this inspires other developers working with NFT data to publicly share their experience and knowledge for the betterment of the community as a whole.Let’s begin by discussing how we selected a source of NFT data…Selecting a data sourceThere were a few considerations that went into choosing the best NFT data API for the NFT Starter Kit:It needs to be freely available (so everybody can try it for free).It needs to provide a diverse set of data points from the NFT ecosystem (to provide a lot of ways to analyze the data).The API server should be reliable in terms of uptime and data quality.OpenSea (and its NFT API), being the largest and one of the first NFT marketplaces, proved to be a great choice for the NFT Starter Kit because it meets all three requirements.About the OpenSea APITheOpenSea APIprovides access to quite a few data fields about assets, accounts, events, and more. Initially when we released the NFT Starter Kit, the OpenSea API didn’t require an API key to use it. Since then, the API does require a key but you can easilyrequest one from their website.The OpenSea API is capable of returning data about events that happened outside the OpenSea platform. This feature enables you to analyze the broader market as well if you want to.Using this API, you can fetch historical and real-time data as well. To be nice to the servers, in the ingestion script we provide, there’s a 2 second delay between requestsby defaultto comfortably fetch historical data from the past 2 years but still relatively fast.It’s also possible to configure the NFT Starter Kitingestion scriptto fetch the most recent data from the past 1 min/hour/day depending on your needs. You just need to modify the time window accordingly in theconfig filein theGitHub repo.Making sense of the API and data fieldsAt the time of creating the NFT Starter Kit (October 2021), the OpenSea documentation felt incomplete. We didn’t get much information from the docs about the data fields we should expect back from the API, or what data type we should use to store those values. But we were still able to explore and learn about the data fields just by experimenting with the API.Most of the data fields were self-explanatory and didn’t require modification before ingesting. There was one field though which was harder to understand for someone new to the crypto space:total_price. One important piece of information which wasn’t documented in the OpenSea API is that thetotal_pricevalue, that was included in each sale event response, should be interpreted asWeiand not ether. This is important because 1 ETH = 1018 Wei. Wei is like satoshi to bitcoin or pennies to the dollar. Hence this value:""total_price"": ""145000000000000000""…is not 145000000000000000 ETH but instead 0.145 ETH. So we needed to convert this value from Wei to ETH before each insert.OpenSea API Python wrapper libraryWe also looked for some API wrapper libraries which are usually already available for popular APIs, but we didn’t find any which covered the /events endpoint. Since encountering this problem, I created andopen sourced anOpenSea API Python wrapperwhich covers all the GET endpoints and makes it easy to fetch NFT data from OpenSea in Python.In general, the OpenSea API is fairly intuitive with a great mechanism for making requests, applying query parameters, and pagination. It also displays explicit error messages in case you do something wrong. The documentation has been improved since we created the NFT Starter Kit, especially as theyadded new pagesfor each data type recently - namely assets, collection, events, and account - that give a brief description of each data field.Next, let’s see how we narrowed our focus to make it easier to ingest NFT data and which endpoints give us relevant time-series data.Focusing on NFT sales dataAs the OpenSea API has multiple interesting endpoints that can provide NFT data, there are many possible ways one could go about analyzing NFTs. From the beginning, we were certain that we wanted to focus on thetime-seriesaspect of the data. Time-series data allows you to analyze NFT trends over time per individual NFT or project for example. Hence, it was natural to use the/eventsendpoint because that endpoint provides data about NFT transactions with timestamps.The JSON output of the OpenSea API /events endpoint (condensed).For a beginner, the complex JSON responses containing loads of data fields provided by this endpoint might be overwhelming at first, so we decided to only focus onsuccessful sale transactionsand ignore offers and other types of events. This also made our schema and data analysis focus more on NFT sales.Focusing on sales data simplified our schema a lot because this way we could freely ignore fields only available foroffers(e.g., bid amount) which would have made us create more tables or make the hypertable more complex.Luckily, the OpenSea API also includes some data fields from other endpoints in the event response. Which means you can fetch not just the event data itself but also additional data about the asset, the collection, and the accounts that participated in the event. So basically the /events endpoint can return four different types of NFT data: asset, collection, accounts (that participated in the sale), and the transaction itself. This allows you to use only one endpoint but still be able to fetch a diverse set of data fields.In order to ingest only successful sale transactions we used theevent_typeOpenSea API URL query parameter and set it to “successful”. This was an important filter because when we started to backfill the data, without any restrictions on event type, we realized that most of the event data wereoffersand not sales. We decided to ignoreoffersdata (for now!) and only focus on sales.If you are interested in not just sale transactions, but other events as well feel free to use the NFT Starter Kit as a starting point and modify it. You canuse our tutorialto get started and then modify the ingest script and schema to ingest not justsalesbutoffersand other event types too so you can analyze those events as well.Next, let’s see how we came up with the final schema design for sales events.Designing an NFT sales schemaGoing through the process of designing a schema can provide a lot of value. Thinking about your schema allows you to start with the questions you want to ask, plan relationships between the data fields, and also formulate the queries you will want to run. It makes you think about what it is that you want to get out of the database. And that’s a very useful step to understanding your data better, even before starting ingesting.In the long run, it might be worthwhile to spend more time at the beginning to design the proper schema so it will be much easier to write efficient queries later on using the relationships and logical connections you determine with the initial schema design.At a high-level, our NFT sales schema needed to achieve three things:It should provide an easy way to query the time-series data andJOINit with relational tables that store additional information (be able to query information about assets and collections (eg. name, link, etc) easily from the time-series table)Have some level of normalization, including foreign keys where necessaryLeave room for future changes regarding what fields we want to analyze deeper (storing “not that interesting” fields in a JSONB column)We tried to come up with a schema that allowed us to make efficient queries involving data fields we found interesting at the time. Other fields still needed to be available but it was okay to store them in a JSONB field.When working with NFT data, if you are not already an NFT and blockchain domain expert, you might not understand what some of the data fields mean. Nonetheless, with some time spent on researching the topic, you can get a decent amount of knowledge to understand the basics. For example, you only havebid_amountvalue for an event if it was an offer and in case of a sale event you have atotal_pricefield instead, or that in a sale transaction thewinner_accountis the one that bought the NFTor there can only be three types of auction, etc.Another aspect is that you need to know what fields need to be (or can be)UNIQUE. For example,collection_slugis unique, there are no two collections with the same slug. Butcollection_nameis not unique.Once you’ve created the correct schema that fits both the domain and your query needs, you can create additional indexes on certain columns.What columns should have indexes? Generally, you should put indexes on columns which you will use inWHEREandINNER JOINclauses. Basically, if you will filter a lot on certain columns you should consider creating an index on those. But be aware that indexes take up space and make inserting data slower as well.Due to OpenSea's lack of detailed field-level documentation (which they improved since the creation of the NFT Starter Kit), we relied on our hands-on experiments using the API and the JSON responses we received, to get a deeper understanding of the different data fields and explore edge cases.Let’s have a look at a snippet from a sample API response:[
   {
       ""approved_account"": null,
       ""asset"": {
           ""id"": 84852466,
           ""image_url"": ""https://lh3.googleusercontent.com/..."",
           ""name"": ""Kangaroo Kingz #41"",
           ""description"": null,
           ""asset_contract"": {...},
           ""permalink"": ""https://opensea.io/assets..."",
           ""collection"": {
               ""description"": ""Come join the Kangaroo Kingz and lets bounce to the moon..."",
               ""name"": ""Kangaroo Kingz""
           },
           ""owner"": {...}
       },
       ""collection_slug"": ""kangaroo-kingz"",
       ""created_date"": ""2021-11-09T07:57:36.108433"",
       ""event_type"": ""successful"",
       ""id"": 1771735968,
       ""quantity"": ""1"",
       ""seller"": {...},
       ""total_price"": ""1600000000000000"",
       ""transaction"": {...},
       ""winner_account"": {...}
   }
]As you can see, we have a lot of interesting data fields in this snippet. One benefit of this kind of response is that it has information not just about the event itself (eg.event_type,quantity,total price) but also other things likeasset,collection,andaccounts. Because of this, you don’t need to make separate requests to fetch additional data, everything is already in this one response.On the other hand, it makes it a little harder to cherry-pick what data fields you really need from this deeply nested JSON object. If you want to generate an example JSON yourself, you can do that in theOpenSea docs.The OpenSea API allows up to 300 items to be included in one response. This allows for faster fetching and ingesting because you can do batches of 300 rather than one-by-one.Storing NFT time-series data in a relational databaseTimescaleDB is a relational time-series database built atop PostgreSQL. It supports full-SQL and you can also JOIN your time-series tables with other regular PostgreSQL tables while having access to time-series specific features.Regular PostgreSQL tables that support the “main” time-series tableTimescaleDB hypertable containing the time-series portion of the dataset.Table definitions and indexesWhen designing a schema that involves time-series data, it’s essential to identify early on which table will store the time-series data. This is important because if you use TimescaleDB that table will need to be converted to a hypertable so it can leverage the underlying TimescaleDB time-based partitioning.You also need to see where it makes sense to use indexes. In general, you want to create indexes on columns that will be often used inWHEREandJOINclauses. In the case of ourNFT Starter Kit tutorialwe decided to put the create index onasset_id,collection_id, andpayment_symbolcolumns to speed up queries where those columns are involved in the filtering. Optionally, you might also want to put an index on one of theaccountcolumns. For example, if you often want to filter bywinner accounts in your queriesit might be worthwhile to create an index onwinner_account. Depends on what you want to focus on in your queries.Note that TimescaleDB hypertables have an index on the time column by default when you create the hypertable, so you don’t need to manually create them.Finally, this became ourschemato store NFT sales data:/* NFT Starter Kit schema definition */
 
CREATE TABLE collections (
   id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
   slug TEXT UNIQUE,
   name TEXT,
   url TEXT,
   details JSONB
);
 
CREATE TABLE accounts (
   id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
   user_name TEXT,
   address TEXT UNIQUE NOT NULL,
   details JSONB
);
 
CREATE TABLE assets (
   id BIGINT PRIMARY KEY,
   name TEXT,
   collection_id BIGINT REFERENCES collections (id), -- collection
   description TEXT,
   contract_date TIMESTAMP WITH TIME ZONE,
   url TEXT UNIQUE,
   img_url TEXT,
   owner_id BIGINT REFERENCES accounts (id), -- account
   details JSONB
);
 
CREATE TYPE auction AS ENUM ('dutch', 'english', 'min_price');
CREATE TABLE nft_sales (
   id BIGINT,
   ""time"" TIMESTAMP WITH TIME ZONE,
   asset_id BIGINT REFERENCES assets (id), -- asset
   collection_id BIGINT REFERENCES collections (id), -- collection
   auction_type auction,
   contract_address TEXT,
   quantity NUMERIC,
   payment_symbol TEXT,
   total_price DOUBLE PRECISION,
   seller_account BIGINT REFERENCES accounts (id), -- account
   from_account BIGINT REFERENCES accounts (id), -- account
   to_account BIGINT REFERENCES accounts (id), -- account
   winner_account BIGINT REFERENCES accounts (id), -- account
   CONSTRAINT id_time_unique UNIQUE (id, time)
);
 
SELECT create_hypertable('nft_sales', 'time');
 
CREATE INDEX idx_asset_id ON nft_sales (asset_id);
CREATE INDEX idx_collection_id ON nft_sales (collection_id);
CREATE INDEX idx_payment_symbol ON nft_sales (payment_symbol);
CREATE INDEX idx_winner_account ON nft_sales (winner_account);Here’s a sample from the nft_sales table:idtimeasset_idcollection_idauction_typecontract_addressquantitypayment_symboltotal_priceseller_accountfrom_accountto_accountwinner_account11617907242021-09-29 11:28:594985850203dutch0xb1690c08e213a35ed9bab7b318de14420fb57d8c1ETH0.00498551108103731511803636584708321470832111686424242021-09-29 22:36:145075728203dutch0xb1690c08e213a35ed9bab7b318de14420fb57d8c1ETH0.0065242098572831811180363658451130845113086533120192021-08-27 03:09:40130356203dutch0xb1690c08e213a35ed9bab7b318de14420fb57d8c1ETH0.005162660293658216428821642889245596572021-09-16 12:04:485179347321625dutch0xb6dae651468e9593e4581705a09c10a76ac1e0c81ETH0.498630903040304013104206362021-10-10 12:13:01626022203dutch0xb1690c08e213a35ed9bab7b318de14420fb57d8c1ETH0.00532521536581285912859NFTs traded in extreme quantitiesIn the initial version of the schema, we usedBIGINTfor the quantity field. Then when we first started ingesting data to test the schema everything looked fine. After a couple tens of thousands of inserted records later we started to get some error messages, like this:psycopg2.errors.NumericValueOutOfRange: value “10000000000000000000” is out of range for type bigintThe data field that caused this issue was the “quantity” field. Unexpectedly there were some NFTs that were traded in extremely high quantities and the value didn’t fit in theBIGINTdata type. So we ended up using theNUMERICPostgreSQL data type which proved to be sufficient. In PostgreSQL numeric and decimal data types are the same, they leave room for up to 131072 digits before the decimal point and up to 16383 digits after the decimal point.Example queriesLet’s look at some example queries from theNFT Starter Kit tutorial.Look up a specific account’s purchase history from the past 3 months/* Snoop Dogg's purchases in the past 3 months aggregated */
WITH snoop_dogg AS (
   SELECT id FROM accounts
   WHERE address = '0xce90a7949bb78892f159f428d0dc23a8e3584d75'
)
SELECT
COUNT(*) AS trade_count,
COUNT(DISTINCT asset_id) AS nft_count,
COUNT(DISTINCT collection_id) AS collection_count,
COUNT(*) FILTER (WHERE winner_account = (SELECT id FROM snoop_dogg)) AS buy_count,
SUM(total_price) AS total_volume_eth,
AVG(total_price) AS avg_price,
MIN(total_price) AS min_price,
MAX(total_price) AS max_price
FROM nft_sales
WHERE payment_symbol = 'ETH' AND winner_account = (SELECT id FROM snoop_dogg)
AND time > NOW()-INTERVAL '3 months'
 
trade_count|nft_count|collection_count|buy_count|total_volume_eth  |avg_price         |min_price|max_price|
-----------+---------+----------------+---------+------------------+------------------+---------+---------+
        58|       57|              20|       58|1825.5040000000006|31.474206896551735|      0.0|   1300.0|This query uses thenft_saleshypertable to return aggregations based on records that involve one specific buyer account (in this example we use Snoop Dogg’s account). The query uses the indexes that we created onwinner_accountandpayment_symbol.Top 5 most expensive NFTs in the CryptoKitties collection/* Top 5 most expensive NFTs in the CryptoKitties collection */
SELECT a.name AS nft, total_price, time, a.url  FROM nft_sales s
INNER JOIN collections c ON c.id = s.collection_id
INNER JOIN assets a ON a.id = s.asset_id
WHERE slug = 'cryptokitties' AND payment_symbol = 'ETH'
ORDER BY total_price DESC
LIMIT 5
 
nft            |total_price|time               |url                                                                    |
---------------+-----------+-------------------+-----------------------------------------------------------------------+
Founder Cat #40|      225.0|2021-09-03 14:59:16|https://opensea.io/assets/0x06012c8cf97bead5deae237070f9587f8e7a266d/40|
Founder Cat #17|      177.0|2021-09-03 01:58:13|https://opensea.io/assets/0x06012c8cf97bead5deae237070f9587f8e7a266d/17|
润龙🐱‍👓创世猫王44# |      150.0|2021-09-03 02:01:11|https://opensea.io/assets/0x06012c8cf97bead5deae237070f9587f8e7a266d/44|
grey           |      149.0|2021-09-03 02:32:26|https://opensea.io/assets/0x06012c8cf97bead5deae237070f9587f8e7a266d/16|
Founder Cat #38|      148.0|2021-09-03 01:58:13|https://opensea.io/assets/0x06012c8cf97bead5deae237070f9587f8e7a266d/38|This query filters thenft_salestable by collection slugandJOINs both thecollectionsandassetstable. The query also uses the indexes that we created on collection_id, payment_symbol, and collections.slug.For more queries check out theGitHub repository, or watch therecording of our live streamwhere you can see the database in action.Resources and learn moreWe hope you found this blog post useful! If you’re interested in trying out the NFT Starter Kit you can do so bycloning our repository on Github. We also suggest following theNFT Analysis tutorial in our docswhich walks you through how to ingest NFT data and also how to analyze it using PostgreSQL and TimescaleDB.The easiest way to store and analyze NFT data (and complete the NFT Starter Kit tutorial!) is using a fully-managed database on Timescale.Sign up here- it’s 100% free for 30-days, no credit card required.👉 Complete the NFT tutorial with Timescale - no credit card required.Sign up for freeIf you prefer managing your own database, you caninstalland use TimescaleDB for free.Furthermore, you can learn more about using TimescaleDB for NFT and other crypto use cases by watching the recordings of theStocks & Crypto SQL Show(join us live onTwitch).If you have questions about the NFT Starter Kit or TimescaleDB in general, join our8,000+ member Slack community, and get help from the Timescale Team and users. You can also tweet your projects and questions to us@TimescaleDBand use the hashtag #TimescaleNFTs.Finally, if you are interested in getting a lot more involved with Timescale and NFT data,we are hiring worldwide!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-store-and-analyze-nft-data-in-a-relational-database/
2023-02-16T15:49:11.000Z,Downsampling in the Database: How Data Locality Can Improve Data Analysis,"Data processing can be a resource-intensive task. And, as datasets grow in size and complexity, the need for efficient data processing and downsampling becomes even more critical, enabling businesses to access information quickly to identify issues and make data-driven decisions early on, ultimately leading to cost and time savings.One way to optimize your data processing is by improving data locality when downsampling your data.Let’s break down these concepts: downsampling data aims to make data more manageable by reducing its granularity. Data locality, on the other hand, is processing data as close as possible to where it’s stored—and nothing can be closer than on the same server.So, in this blog post, we’ll perform calculations close to the data—directly in the database—to optimize data processing. We’ll evaluate its benefits and compare it to processing in the client. Finally, we’ll exploreTimescale hyperfunctionsas an alternative so we don’t have to pull data from a database to an analytics framework hosted in your programming language.Downsampling Decisions and My Path to Data LocalityDevelopers make decisions about downsampling and where to process data every day, whether or not they realize it. I remember my early days as a Ruby developer when I always tried to write beautiful code—I wanted it to look like poetry. I didn’t worry too much about the most efficient way to do things, but I strived to achieve good readability and meet business requirements.Moving to the financial markets sector represented a huge shift. I started building trading systems in which speed was a requirement, and after days and months of optimizing code for faster trading decisions, I started seeing things very differently. Later, I joined a fast-growing company to use my code-optimization skills and carefully review several services and background jobs fighting for resources.My perspective on expensive processing architecture expanded, as my job’s most common task was benchmarking and checking metrics around resource consumption. Every move was made to optimize the system to become more scalable: my career went from carefree coding to explicitly benchmarking my choices and analyzing the performance of each refactoring or improvement.My competence as a programmer grew alongside the systems I worked on. I deepened my knowledge about how programming languages can use resources behind the scenes. Now, my internal checklist still looks for performance issues, and, as a user, I celebrate well-performing systems.So, I am well aware that as datasets grow in size and complexity, the trade-offs associated with data processing and downsampling become more critical. Engineering departments welcome data teams and analysts, and eventually, the hardware infrastructure is expanded to supply the new demand.With engineers carefully analyzing the infrastructure costs and metrics, they identify bottlenecks. Input/output (I/O) blocking is a common misusage that can lead to idle threads waiting for the network while other threads starve or deadlock. As the system scales for more users, new resources can unbalance, and you need a new round of architecture review not to depreciate the user experience.That’s wherelatencycomes into play.LatencyLatency refers to the time it takes for information to travel from one point in a system resource to another. It includes bothnetwork latency(the time it takes for data to travel over a network) andprocessing latency(the time it takes for information to be processed by an application).By performing calculations close to the data and downsampling in the database, you can reduce overall latency, positively impacting the user experience and the overall system performance metrics, such as response time or throughput rates.It also helps to keep your data clean and consistent across different nodes in your distributed system.But first, let’s see how data travels.How Data TravelsIt’s almost impossible to list all the computational steps involved in data processing that are hidden and often forgotten by developers, from querying in the database to visualizing data in the system.Summing it up, everything starts with the low-level steps the database engine takes to bring data tuples from a SQL query.First, the parser reads your query and transforms it into an abstract syntax tree. If there’s room for simplification, the tree is simplified to make the routes more accessible. The query tree is a set of nodes and child nodes representing paths and constraints where the data comes from, almost like a travel plan.You can fetch data in many ways, and the planner calculates the cheapest route to get the target data. A route plan is chosen in a few milliseconds, and the executor takes over the cheapest plan to fetch the data. When the data tuples are in the memory, a serializing process allows the data to transit in the network.The network talks in its own protocol, which wraps the data into network packets shielded with security layers. At the speed of light, the data packets are safely crossed under the oceans through optical fibers, choosing the most optimized routes to arrive as early as possible in the target machine. Crossing a continent can cost a few hundred milliseconds.The data arrival in the target processing unit also takes some extra time: data passes through “boarding gates” to confirm that its identity is trustable, its packet consistency is verified, and finally, if it’s still relevant, the unpacking process takes place. With unpacked data in the I/O, the data is routed to the database client driver before the deserialization process starts loading the data in the processing unit, which is another programming language.We'd need to put the programming language layer under another microscope to analyze how the data is streamed to the context. Indeed, data is not alone. Taking Ruby as an example, it would often arrive in an expensive processing context, bloated by frameworks and other runtime facilities optimized for developers’ productivity but overlooking the computational resources.Each programming language has its own focus and benefits. TheEnergy Efficiency across Programming Languagesstudy can help you dive deeper into the topic. “Table 4. Normalized global results for Energy, Time, and Memory” unifies several benchmark scenarios that can inspire you to make environmentally friendly choices. You'll notice that C and Rust, in which PostgreSQL and hyperfunctions are written, are incredibly energy efficient—they can do more while consuming fewer CPU cycles.SourceDownsampling in the Database: How the Timescale Hyperfunctions Can HelpWe createdhyperfunctions(which are provided by theTimescaleDB Toolkitextension and come already pre-installed into Timescale) to bring data analysis superpowers to SQL. Increasing data processing power and speed can help you process significant amounts of data more quickly and accurately.Hyperfunctions are a set of mathematical functions and algorithms written in Rust and packaged as a PostgreSQL extension. They work incredibly well with Timescale and TimescaleDB, aiming to increase data processing efficiency with a first-class PostgreSQL experience.The bounded functions are available in the database and can be combined with standard SQL queries. So, instead of allocating resources from your system application and competing with users trying to access the system,data processing can be delegated to the database and easily scaled with database replicas.Adding data processing features to the application layer can make the development phase challenging, messing with other development tasks and the deployment of features. Moving the analysis to the database will allow more flexibility to the data analysts and relieve the application workload, thus releasing I/O waiting time with fewer context-switching threads. Finally, it will also reduce the network traffic between the application and the database.The challenge is more about changing the team’s mindset than adding a new layer of complexity.Comparing Downsampling Data in Ruby vs. TimescaleIt’s time to jump to a practical scenario. Let’s see how data processing in Ruby compares to doing it in the database—analyzing how the processing decisions can affect the application performance and user experience.The scenario we’re testing here is theLargest Triangle Three Bucketsfunction, a downsampling method that tries to retain visual similarity between the downsampled data and the original dataset. If you’re curious about the LTTB algorithm and its Ruby implementation, check theLTTB tutorialthat implements the Ruby code from scratch.While most frameworks implement the code in the front end, Timescale provides an implementation that takes (timestamp, value) pairs, sorts them if needed, and downsamples the values directly in the database.The following animation shows Ruby and SQL endpoints downsampling 400,000 records from an open weather dataset—downsampling 40 years of Vienna temperatures to 500 points for a readable visualization. The data is displayed as it becomes available.Regardless of using Ruby, downsampling the pair values alone would save approximately 13 MB in the network, resulting in 17 KB, which is less than 1 % of the original request. By reducing the amount of data we send we also reduce the amount of data we process at the other end, reducing CPU and memory requirements as well.When we put it all together the SQL version powered by hyperfunctions uses only about 17 % of the time compared to Ruby, considering Ruby took 3.5 seconds of processing time to downsample 400,000 points to 500 points. If you were patient enough to see the GIF until the end, you could see that the zoom feature refreshed and reloaded 500 points again to keep the exact data resolution.With less data, it will reduce the discrepancy between both sides.The more data your system stores, the more advantageous it will be to process it closer to the database.CPU Bound vs. I/O BoundThe choice of downsampling or not the data in the database will always be a trade-off between CPU and I/O. If you choose to process data on your database, it will still process and delegate the work to the database. Depending on your application’s bound, you will better balance your resources by splitting the work and delegating some processing to the database.This means that if your database is working solely like a storage layer and dedicated to persisting data, there’s a high probability that the CPU and memory are underused, and you could leverage a bit of the hardware utilization with this method.Plus, depending on the requirements of your system, you can process data on replicas with minimal impact on the primary database.Timescale + Hyperfunctions = Dizzying Downsampling SpeedThe Timescale Team is busy carefully designing hyperfunctions to increase the time-series superpowers that Timescale already delivers to vanilla PostgreSQL.Compression,table partitioning, andcontinuous aggregateswork with most hyperfunctions. Also, several are already parallel-safe, leveraging hypertable chunks to process data even faster.In conclusion, we’ve seen how data processing can be resource-intensive, and its efficiency is becoming increasingly crucial as datasets grow in size and complexity. Successful businesses will need to do it fast to accelerate data-driven decisions.Timescale hyperfunctions fill this gap with a large set of statistical functions to empower your data processing layer, allowing better data locality as your calculations get closer to the data itself.We’ve explored the benefits of downsampling in the database compared to processing in Ruby. We’ve also shown how Timescale optimized hyperfunctions for large datasets and parallel safety. Bringing the data processing closer to the database will not only relieve your system resources but also save you money and give you more and happier users with the same computational power.We hope we’ve opened your mind to this approach. If you want to try it out,get going quickly with Timescale, our cloud database platform (30-day free trial, no credit card required), andexplore hyperfunctions with TimescaleDB.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/downsampling-in-the-database-how-data-locality-can-improve-data-analysis/
2022-06-21T12:58:38.000Z,How We Made Data Aggregation Better and Faster on PostgreSQL With TimescaleDB 2.7,"It’s time for another #AlwaysBeLaunching week! 🥳🚀✨ In our #AlwaysBeLaunching initiatives, we challenge ourselves to bring you an array of new features and content. Today, we are introducing TimescaleDB 2.7 and the performance boost it brings for aggregate queries. 🔥 Expect more news this week about further performance improvements, developer productivity, SQL, and more. Make sure you follow us on Twitter (@TimescaleDB), so you don’t miss any of it!Time-series datais the lifeblood of the analytics revolution in nearly every industry today. One of the most difficult challenges for application developers and data scientists is aggregating data efficiently without always having to query billions (or trillions) of raw data rows. Over the years, developers and databases have created numerous ways to solve this problem, usually similar to one of the following options:DIY processes to pre-aggregate data and store it in regular tables. Although this provides a lot of flexibility, particularly with indexing and data retention, it's cumbersome to develop and maintain, particularly deciding how to track and update aggregates with data that arrives late or has been updated in the past.Extract Transform and Load (ETL) process for longer-term analytics.Even today, development teams employ entire groups that specifically manage ETL processes for databases and applications because of the constant overhead of creating and maintaining the perfect process.MATERIALIZED VIEWS.While these VIEWS are flexible and easy to create, they are static snapshots of the aggregated data. Unfortunately, developers need to manage updates using TRIGGERs or CRON-like applications in all current implementations. And in all but a very few databases, all historical data is replaced each time, preventing developers from dropping older raw data to save space and computation resources every time the data is refreshed.Most developers head down one of these paths because we learn, often the hard way, that running reports and analytic queries over the same raw data, request after request, doesn't perform well under heavy load. In truth, most raw time-series data doesn't change after it's been saved, so these complex aggregate calculations return the same results each time.In fact, as a long-term time-series database developer, I've used all of these methods too, so that I could manage historical aggregate data to make reporting, dashboards, and analytics faster and more valuable, even under heavy usage.I loved when customers were happy, even if it meant a significant amount of work behind the scenes maintaining that data.But, I always wished for a more straightforward solution.How TimescaleDB Improves Queries on Aggregated Data in PostgreSQLIn 2019, TimescaleDB introduced continuous aggregates to solve this very problem, making the ongoing aggregation of massive time-series data easy and flexible. This is the feature that first caught my attention as a PostgreSQL developer looking to build more scalable time-series applications—precisely because I had been doing it the hard way for so long.Continuous aggregates look and act like materialized views in PostgreSQL, but with many of the additional features I was looking for (if you want to learn more about views, materialized views, and continuous aggregates, check out this lesson from our Foundations of PostgreSQL and TimescaleDB course). These are just some of the things they do:Automatically track changes and additions to the underlying raw data.Provide configurable, user-defined policies to keep the materialized data up-to-date automatically.Automatically append new data (asreal-time aggregatesby default) before the scheduled process has materialized to disk. This setting is configurable.Retain historical aggregated data even if the underlying raw data is dropped.Can be compressed to reduce storage needs and further improve the performance of analytic queries.Keep dashboards and reports running smoothly.Table comparing the functionality of PostgreSQL materialized views with continuous aggregates in TimescaleDBOnce I tried continuous aggregates, I realized that TimescaleDB provided the solution that I (and many other PostgreSQL users) were looking for. With this feature, managing and analyzing massive volumes of time-series data in PostgreSQL finally felt fast and easy.What About Other Databases?By now, some readers might be thinking something along these lines:“Continuous aggregates may help with the management and analytics of time-series data in PostgreSQL, but that’s what NoSQL databases are for—they already provide the features you needed from the get-go. Why didn’t you try a NoSQL database?”Well, I did.There are numerous time-series and NoSQL databases on the market that attempt to solve this specific problem. I looked at (and used) many of them. But from my experience, nothing can quite match the advantages of a relational database with a feature like continuous aggregates for time-series data. These other options provide a lot of features for a myriad of use cases, but they weren't the right solution for this particular problem, among other things.What about MongoDB?MongoDBhas been the go-to for many data-intensive applications. Included since version 4.2 is a feature calledOn-Demand Materialized Views. On the surface, it works similar to a materialized view by combining theAggregation Pipelinefeature with a $merge operation to mimic ongoing updates to an aggregate data collection. However, there is no built-in automation for this process, and MongoDB doesn't keep track of any modifications to underlying data. The developer is still required to keep track of which time frames to materialize and how far back to look.What about InfluxDB?For many yearsInfluxDBhas been the destination for time-series applications. Althoughwe've discussed in other articles how InfluxDB doesn't scale effectively, particularly with high cardinality datasets, it does provide a feature calledContinuous Queries.This feature is also similar to a materialized view and goes one step further than MongoDB by automatically keeping the dataset updated. Unfortunately, it suffers from the same lack of raw data monitoring and doesn't provide nearly as much flexibility as SQL in how the datasets are created and stored.What about Clickhouse?Clickhouse, and several recent forks likeFirebolt, have redefined the way some analytic workloads perform. Even with some of theimpressive query performance, it provides a mechanism similar to a materialized view as well, backed by anAggregationMergeTreeengine. In a sense, this provides almost real-time aggregated data because all inserts are saved to both the regular table and the materialized view. The biggest downside of this approach is dealing with updates or modifying the timing of the process.Recent Improvements in Continuous Aggregates: Meet TimescaleDB 2.7Continuous aggregates were first introduced inTimescaleDB 1.3solving the problems that many PostgreSQL users, including me, faced with time-series data and materialized views: automatic updates, real-time results, easy data management, and the option of using the view for downsampling.But continuous aggregates have come a long way. One of the previous improvements was the introduction ofcompression for continuous aggregates in TimescaleDB 2.6. Now, we took it a step further with the arrival of TimescaleDB 2.7, which introduces dramatic performance improvements in continuous aggregates.They are now blazing fast—up to 44,000x faster in some queries than in previous versions.Let me give you one concrete example:in initial testing using live, real-time stock trade transaction data, typical candlestick aggregates were nearly 2,800x faster to querythan in previous versions of continuous aggregates (which were already fast!)Later in this post, we will dig into the performance and storage improvements introduced by TimescaleDB 2.7 by presenting a complete benchmark of continuous aggregates using multiple datasets and queries. 🔥But the improvements don’t end here.First, the new continuous aggregates also require 60 % less storage (on average) than before for many common aggregates, which directly translates into storage savings.Second, in previous versions of TimescaleDB, continuous aggregates came with certain limitations: users, for example, could not use certain functions like DISTINCT, FILTER, or ORDER BY. These limitations are now gone. TimescaleDB 2.7 ships with a completely redesigned materialization process that solves many of the previous usability issues, so you can use any aggregate function to define your continuous aggregate.Check out our release notes for all the details on what's new.✨ A big thank you to the Timescale engineers that made the improvements in continuous aggregates possible, with special mentions to Fabrízio Mello, Markos Fountoulakis, and David Kohn.And now, the fun part.Show Me the Numbers: Benchmarking Aggregate QueriesTo test the new version of continuous aggregates, we chose two datasets that represent common time-series datasets: IoT and financial analysis.IoT dataset (~1.7 billion rows):The IoT data we leveraged is the New York City Taxicab dataset that's been maintained by Todd Schneider for a number of years, and scripts are available in hisGitHub repositoryto load data into PostgreSQL. Unfortunately, a week after his latest update, the transit authority that maintains the actual datasets changed their long-standing export data format from CSV to Parquet—which means the current scripts will not work. Therefore, the dataset we tested with is from data prior to that change and covers ride information from 2014 to 2021.Stock transactions dataset (~23.7 million rows):The financial dataset we used is a real-time stock trade dataset provided byTwelve Dataand ingests ongoing transactions for the top 100 stocks by volume from February 2022 until now. Real-time transaction data is typically the source of many stock trading analysis applications requiring aggregate rollups over intervals for visualizations likecandlestick chartsand machine learning analysis. While our example dataset is smaller than a full-fledged financial application would maintain, it provides a working example of ongoing data ingestion using continuous aggregates, TimescaleDBnative compression, andautomated raw data retention(while keeping aggregate data for long-term analysis).You can use a sample of this data, generously provided by Twelve Data, to try all of the improvements in TimescaleDB 2.7 by followingthis tutorial, which provides stock trade data for the last 30 days. Once you have the database setup, you can take it a step further by registering for an API key andfollowing our tutorial to ingest ongoing transactions from the Twelve Data API.Creating Continuous Aggregates Using Standard PostgreSQL Aggregate FunctionsThe first thing we benchmarked was to create an aggregate query that used standard PostgreSQL aggregate functions likeMIN(),MAX(), andAVG(). In each dataset we tested, we created the same continuous aggregate in TimescaleDB 2.6.1 and 2.7, ensuring that both aggregates had computed and stored the same number of rows.IoT datasetThis continuous aggregate resulted in 1,760,000 rows of aggregated data spanning seven years of data.CREATE MATERIALIZED VIEW hourly_trip_stats
WITH (timescaledb.continuous, timescaledb.finalized=false) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
	avg(fare_amount) avg_fare,
	min(fare_amount) min_fare,
	max(fare_amount) max_fare,
	avg(trip_distance) avg_distance,
	min(trip_distance) min_distance,
	max(trip_distance) max_distance,
	avg(congestion_surcharge) avg_surcharge,
	min(congestion_surcharge) min_surcharge,
	max(congestion_surcharge) max_surcharge,
	cab_type_id,
	passenger_count
FROM 
	trips
GROUP BY 
	bucket, cab_type_id, passenger_countStock transactions datasetThis continuous aggregate resulted in 950,000 rows of data at the time of testing, although these are updated as new data comes in.CREATE MATERIALIZED VIEW five_minute_candle_delta
WITH (timescaledb.continuous) AS
    SELECT
        time_bucket('5 minute', time) AS bucket,
        symbol,
        FIRST(price, time) AS ""open"",
        MAX(price) AS high,
        MIN(price) AS low,
        LAST(price, time) AS ""close"",
        MAX(day_volume) AS day_volume,
        (LAST(price, time)-FIRST(price, time))/FIRST(price, time) AS change_pct
    FROM stocks_real_time srt
    GROUP BY bucket, symbol;To test the performance of these two continuous aggregates, we selected the following queries, all common queries among our users for both the IoT and financial use cases:SELECT COUNT (*)SELECT COUNT (*) with WHEREORDER BYtime_bucket reaggregationFILTERHAVINGLet’s take a look at the results.Query #1: `SELECT COUNT(*) FROM…`Doing aCOUNT(*)from PostgreSQL is a known performance bottleneck. It's one of the reasons we created theapproximate_row_count()function in TimescaleDB which uses table statistics to provide a close approximation of the overall row count. However, it's instinctual for most users (and ourselves, if we're honest) to try and get a quick row count by doing aCOUNT(*)query:-- IoT dataset
SELECT count(*) FROM hourly_trip_stats;

-- Stock transactions dataset
SELECT count(*) FROM five_min_candle_delta;And most users recognized that in previous versions of TimescaleDB, the materialized data seemed slower than normal to do a COUNT over.Thinking about our two example datasets, both continuous aggregates reduce the overall row count from raw data by 20x or more. So, while counting rows in PostgreSQL is slow, it always felt a little slower than it had to be. The reason was that not only did PostgreSQL have to scan and count all of the rows of data, it had to group the data a second time because of some additional data that TimescaleDB stored as part of the original design of continuous aggregates. With the new design of continuous aggregates in TimescaleDB 2.7, that second grouping is no longer required, and PostgreSQL can just query the data normally, translating into faster queries.Performance of a query with SELECT COUNT (*) in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7Query #2: SELECT COUNT(*) Based on The Value of a ColumnAnother common query that many analytic applications perform is to count the number of records where the aggregate value is within a certain range:-- IoT  dataset
SELECT count(*) FROM hourly_trip_stats
WHERE avg_fare > 13.1
AND bucket > '2018-01-01' AND bucket < '2019-01-01';

-- Stock transactions dataset
SELECT count(*) FROM five_min_candle_delta
WHERE change_pct > 0.02;In previous versions of continuous aggregates, TimescaleDB had to finalize the value before it could be filtered against the predicate value, which caused queries to perform more slowly. With the new version of continuous aggregates, PostgreSQL can now search for the value directly,andwe can add an index to meaningful columns to speed up the query even more!In the case of the financial dataset, we see a very significant improvement: 1,336x faster. The large change in performance can be attributed to the formula query that has to be calculated over all of the rows of data in the continuous aggregate. With the IoT dataset, we're comparing against a simple average function, but for the stock data, multiple values have to be finalized (FIRST/LAST) before the formula can be calculated and used for the filter.Performance of a query with SELECT COUNT (*) plus WHERE in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7Query #3: Select Top 10 Rows by ValueTaking the first example a step further, it's very common to query data within a range of time and get the top rows:-- IoT dataset
SELECT * FROM hourly_trip_stats
ORDER BY avg_fare desc
LIMIT 10;

-- Stock transactions dataset
SELECT * FROM five_min_candle_delta
ORDER BY change_pct DESC 
LIMIT 10;In this case, we tested queries with the continuous aggregate set to providereal-time results(the default for continuous aggregates) and materialized-only results. When set to real-time, TimescaleDB always queries data that's been materialized first and then appends (with aUNION) any newer data that exists in the raw data but that has not yet been materialized by the ongoing refresh policy. And, because it's now possible to index columns within the continuous aggregate, we added an index on theORDER BYcolumn.Performance of a query with ORDER BY in a continuous aggregate TimescaleDB 2.6.1 and TimescaleDB 2.7Yes, you read that correctly.Nearly 45,000x better performance onORDER BYwhen the query only searches through materialized data.The dramatic difference between real-time and materialized-only queries is because of theUNIONof both materialized and raw aggregate data. The PostgreSQL planner needs to union the total result before it can limit the query to 10 rows (in our example), and so all of the data from both tables need to be read and ordered first. When you only query materialized data, PostgreSQL and TimescaleDB knows that it can query just the index of the materialized data.Again, storing the finalized form of your data and indexing column values dramatically impacts the querying performance of historical aggregate data! And all of this is updated continuously over time in a non-destructive way—something that's impossible to do with any other relational database, including vanilla PostgreSQL.Query #4: Timescale Hyperfunctions to Re-aggregate Into Higher Time BucketsAnother example we wanted to test was the impact finalizing data values has on our suite ofanalytical hyperfunctions. Many of the hyperfunctions we provide as part of theTimescaleDB Toolkitutilize custom aggregate values that allow many different values to be accessed later depending on the needs of an application or report. Furthermore, these aggregate values can bere-aggregated into different size time buckets. This means that if the aggregate functions fit your use case, one continuous aggregate can produce results for many different time_bucket sizes! This is a feature many users have asked for over time, and hyperfunctions make this possible.For this example, we only examined the New York City Taxicab dataset to benchmark the impact of finalized CAGGs. Currently, there is not an aggregate hyperfunction that aligns with the OHLC values needed for the stock data set, however,there is a feature requestfor it! (😉)Although there are not currently any one-to-one hyperfunctions that provide exact replacements for our min/max/avg example, we can still observe the query improvement using atdigestvalue for each of the columns in our original query.Original min/max/avg continuous aggregate for multiple columns:CREATE MATERIALIZED VIEW hourly_trip_stats
WITH (timescaledb.continuous, timescaledb.finalized=false) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
	avg(fare_amount) avg_fare,
	min(fare_amount) min_fare,
	max(fare_amount) max_fare,
	avg(trip_distance) avg_distance,
	min(trip_distance) min_distance,
	max(trip_distance) max_distance,
	avg(congestion_surcharge) avg_surcharge,
	min(congestion_surcharge) min_surcharge,
	max(congestion_surcharge) max_surcharge,
	cab_type_id,
	passenger_count
FROM 
	trips
GROUP BY 
	bucket, cab_type_id, passenger_countHyperfunction-based continuous aggregate for multiple columns:CREATE MATERIALIZED VIEW hourly_trip_stats_toolkit
WITH (timescaledb.continuous, timescaledb.finalized=false) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
	tdigest(1,fare_amount) fare_digest,
	tdigest(1,trip_distance) distance_digest,
	tdigest(1,congestion_surcharge) surcharge_digest,
	cab_type_id,
	passenger_count
FROM 
	trips
GROUP BY 
	bucket, cab_type_id, passenger_countWith the continuous aggregate created, we then queried this data in two different ways:1. Using the same `time_bucket()` size defined in the continuous aggregate, which in this example was one-hour data.SELECT 
	bucket AS b,
	cab_type_id, 
	passenger_count,
	min_val(ROLLUP(fare_digest)),
	max_val(ROLLUP(fare_digest)),
	mean(ROLLUP(fare_digest))
FROM hourly_trip_stats_toolkit
WHERE bucket > '2021-05-01' AND bucket < '2021-06-01'
GROUP BY b, cab_type_id, passenger_count 
ORDER BY b DESC, cab_type_id, passenger_count;Performance of a query with time_bucket() in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7 (the query uses the same bucket size as the definition of the continuous aggregate)2. We re-aggregated the data from one-hour buckets into one-day buckets.This allows us to efficiently query different bucket lengths based on the original bucket size of the continuous aggregate.SELECT 
	time_bucket('1 day', bucket) AS b,
	cab_type_id, 
	passenger_count,
	min_val(ROLLUP(fare_digest)),
	max_val(ROLLUP(fare_digest)),
	mean(ROLLUP(fare_digest))
FROM hourly_trip_stats_toolkit
WHERE bucket > '2021-05-01' AND bucket < '2021-06-01'
GROUP BY b, cab_type_id, passenger_count 
ORDER BY b DESC, cab_type_id, passenger_count;Performance of a query with time_bucket() in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7. The query re-aggregates the data from one-hour buckets into one-day bucketsIn this case, the speed is almost identical because the same amount of data has to be queried. But if these aggregates satisfy your data requirements, only one continuous aggregate would be necessary in many cases, rather than a different continuous aggregate for each bucket size (one minute, five minutes, one hour, etc.)Query #5: Pivot Queries With FILTERIn previous versions of continuous aggregates, many common SQL features were not permittedbecause of how the partial data was stored and finalized later. Using a PostgreSQLFILTERclause was one such restriction.For example, we took the IoT dataset and created a simpleCOUNT(*)to calculate each company's number of taxi rides (cab_type_id) for each hour. Before TimescaleDB 2.7, you would have to store this data in a narrow column format, storing a row in the continuous aggregate for each cab type.CREATE MATERIALIZED VIEW hourly_ride_counts_by_type 
WITH (timescaledb.continuous, timescaledb.finalized=false) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
	cab_type_id,
  	COUNT(*)
FROM trips
  	WHERE cab_type_id IN (1,2)
GROUP BY 
	bucket, cab_type_id;To then query this data in a pivoted fashion, we couldFILTERthe continuous aggregate data after the fact.SELECT bucket,
	sum(count) FILTER (WHERE cab_type_id IN (1)) yellow_cab_count,
  	sum(count) FILTER (WHERE cab_type_id IN (2)) green_cab_count
FROM hourly_ride_counts_by_type
WHERE bucket > '2021-05-01' AND bucket < '2021-06-01'
GROUP BY bucket
ORDER BY bucket;In TimescaleDB 2.7, you can now store the aggregated data using aFILTERclause to achieve the same result in one step!CREATE MATERIALIZED VIEW hourly_ride_counts_by_type_new 
WITH (timescaledb.continuous) 
AS
SELECT 
	time_bucket('1 hour',pickup_datetime) bucket,
  	COUNT(*) FILTER (WHERE cab_type_id IN (1)) yellow_cab_count,
  	COUNT(*) FILTER (WHERE cab_type_id IN (2)) green_cab_count
FROM trips
GROUP BY 
	bucket;Querying this data is much simpler, too, because the data is already pivoted and finalized.SELECT * FROM hourly_ride_counts_by_type_new 
WHERE bucket > '2021-05-01' AND bucket < '2021-06-01'
ORDER BY bucket;This saves storage (50 % fewer rows in this case) and CPU to finalize theCOUNT(*)and then filter the results each time based oncab_type_id. We can see this in the query performance numbers.Performance of a query with FILTER in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7.Being able to useFILTERand other SQL features improve both developer experience and flexibility long term!Query #6: HAVING Stores Significantly Less Materialized DataAs a final example of how the improvements to continuous aggregates will impact your day-to-day development and analytics processes, let's look at a simple query that uses aHAVINGclause to reduce the number of rows that the aggregate stores.In previous versions of TimescaleDB, the having clause couldn't be applied at materialization time. Instead, theHAVINGclause was applied after the fact to all of the aggregated data as it was finalized. In many cases, this dramatically affected both the speed of queries to the continuous aggregate and the amount of data stored overall.Using our stock data as an example, let's create a continuous aggregate that only stores a row of data if thechange_pctvalue is greater than 20 %. This would indicate that a stock price changed dramatically over one hour, something we don't expect to see in most hourly stock trades.CREATE MATERIALIZED VIEW one_hour_outliers
WITH (timescaledb.continuous) AS
    SELECT
        time_bucket('1 hour', time) AS bucket,
        symbol,
        FIRST(price, time) AS ""open"",
        MAX(price) AS high,
        MIN(price) AS low,
        LAST(price, time) AS ""close"",
        MAX(day_volume) AS day_volume,
        (LAST(price, time)-FIRST(price, time))/LAST(price, time) AS change_pct
    FROM stocks_real_time srt
    GROUP BY bucket, symbol
   HAVING (LAST(price, time)-FIRST(price, time))/LAST(price, time) > .02;Once the dataset is created, we can query each aggregate to see how many rows matched our criteria.SELECT count(*) FROM one_hour_outliers;Performance of a query with HAVING in a continuous aggregate in TimescaleDB 2.6.1 and TimescaleDB 2.7The biggest difference here (and the one that will more negatively impact the performance of your application over time) is the storage size of this aggregated data. Because TimescaleDB 2.7 only stores rows that meet the criteria, the data footprint is significantly smaller!Storage footprint of a continuous aggregate bucketing stock transactions by the hour in TimescaleDB 2.6.1 and TimescaleDB 2.7Storage Savings in TimescaleDB 2.7One of the final pieces of this update that excites us is how much storage will be saved over time. On many occasions, users with large datasets that contained complex equations in their continuous aggregates would join ourSlack communityto ask why more storage is required for the rolled-up aggregate than the raw data.In every case we've tested, the new, finalized form of continuous aggregates is smaller than the same example in previous versions of TimescaleDB, with or without aHAVINGclause that might filter additional data out.Storage savings for different continuous aggregates in TimescaleDB 2.6.1 and TimescaleDB 2.7The New Continuous Aggregates Are a Game-ChangerFor those dealing with massive amounts of time-series data, continuous aggregates are the best way to solve a problem that has long haunted PostgreSQL users. The following list details how continuous aggregates expand materialized views:They always stay up-to-date, automatically tracking changes in the source table for targeted, efficient updates of materialized data.You can use configurable policies to conveniently manage refresh/update interval.You can keep your materialized data even after the raw data is dropped, allowing you to downsample your large datasets.And you can compress older data to save space and improve analytic queries.And in TimescaleDB 2.7, continuous aggregates got much better. First, they are blazing fast: as we demonstrated with our benchmark, the performance of continuous aggregates got consistently better across queries and datasets, up to thousands of times better for common queries. They also got lighter, requiring an average of 60 % less storage.But besides the performance improvements and storage savings, there are significantly fewer limitations on the types of aggregate queries you can use with continuous aggregates, such as:Aggregates with DISTINCTAggregates with FILTERAggregates with FILTER in HAVING clauseAggregates without combine functionOrdered-set aggregatesHypothetical-set aggregatesThis new version of continuous aggregates is available by default inTimescaleDB 2.7: now, when you create a new continuous aggregate, you will automatically benefit from all the latest changes.Read our release notes for more information on TimescaleDB 2.7, and for instructions on how to upgrade,check out our docs.Looking to migrate your existing continue aggregates to the new version? Now, with TimescaleDB 2.8.1, you don’t have to worry aboutmigrating from the old continuous aggregates to the new. Say hello to our frictionless migration, an in-place upgrade that avoids disrupting queries over continuous aggregates in applications and dashboards and every time the data is not in the original hypertable.☁️🐯 Timescale Cloud avoids the manual work involved in updating your TimescaleDB version. Updates take place automatically during a maintenance window picked by you.Learn moreabout automatic version updates in Timescale Cloud, and to test if yourself,start a free trial!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-we-made-data-aggregation-better-and-faster-on-postgresql-with-timescaledb-2-7/
2022-09-22T15:32:44.000Z,"PostgreSQL + TimescaleDB: 1,000x Faster Queries, 90 % Data Compression, and Much More","Compared to PostgreSQL alone, TimescaleDB can dramatically improve query performance by 1000x or more, reduce storage utilization by 90 %, and provide features essential for time-series and analytical applications. Some of these features even benefit non-time-series data–increasing query performance just by loading the extension.PostgreSQL is today’s most advanced and most popular open-source relational database. We believe this as much today as we didfive years ago, when we chose PostgreSQL as the foundation of TimescaleDB because of its longevity, extensibility, and rock-solid architecture.By loading the TimescaleDB extension into a PostgreSQL database, you can effectively “supercharge” PostgreSQL, empowering it to excel for both time-series workloads and classic transactional ones.This article highlights how TimescaleDB improves PostgreSQL query performance at scale, increases storage efficiency (thus lowering costs), and provides developers with the tools necessary for building modern, innovative, and cost-effective time-series applications – all while retaining access to the full Postgres feature-set and ecosystem.(To show our work this article also presents the benchmarks that compare query performance and data ingestion for 1 billion rows of time-series data between PostgreSQL 14.4 and TimescaleDB 2.7.2.  For PostgreSQL, we benchmarked both using a single-table and declarative partitioning)Better Performance at ScaleWith orders of magnitude better performance at scale, TimescaleDB enables developers to build on top of PostgreSQLand“future-proof” their applications.1,000x Faster Performance for Time-series QueriesThe core concept in TimescaleDB is the notion of the “hypertable”: seamless partitioning of data, while presenting the abstraction of a single, virtual table across all your data.This partitioning enables faster queries by quickly excluding irrelevant data, as well as enabling enhancements to the query planner and execution process. In this way, a hypertable looks and feels just like a normal PostgreSQL table but enables a lot more.For example, one recent query planner improvement excludes data more efficiently for relativenow()-based queries (e.g.,WHERE time >= now()-’1 week’::interval). To be even more specific,these relative time predicates are constifiedat planning time to ignore chunks that don't have data to satisfy the query. Furthermore, as the number of partitions increases, planning times can be reduced by 100x or more over vanilla PostgreSQL for the same number of partitions.When hypertables are compressed the amount of data that queries need to read is reduced, leading to dramatic increases in performance of 1000x or more. For more information (including a discussion of this bar chart) keep reading to the benchmark below.Query latency comparison (ms) between TimescaleDB and PostgreSQL 14.4. To see the complete query, scroll down below.Other enhancements in TimescaleDB apply to both hypertables and normal PostgreSQL tables: e..g, SkipScan, whichdramatically improves DISTINCT queries on any PostgreSQL tablewith a matching B-tree index regardless of if you have time-series data or not.Reduce Commonly Run Queries to Milliseconds (Even When The Original Query Took Minutes or Hours)Today nearly every time-series application reaches for rolling aggregations to query and analyze data more efficiently. The raw data could be saved per second, minute, or hour (and a plethora of other permutations in between), but what most applications display are time-based aggregates.What's more, most time-series data applications are append-only, which means that aggregate queries return the same values over and over based on the unchanged raw data. It's much more efficient to store the results of the aggregate query and use those for analytic reporting and analysis most of the time.Often developers try materialized views in vanilla PostgreSQL to help, however, they have two main problems with fast-changing time-series data:Materialized viewsrecreate the entire view every time the materialization process runs,even if little or no data has changed.Materialized views don't provide any data retention management. Any time you delete raw data and update the materialized view, the aggregated data is removed as well.In contrast,TimescaleDB’s continuous aggregatessolve both of these problems. They are updated automatically on theschedule you configure, they can have dataretention policies applied separately from the underlying hypertable, and they only update the portions of new data that have been modified since the last materialization was run.When we compare using a continuous aggregate to querying the data directly, customers often see queries which might take minutes or even hours drop to milliseconds. When that query is powering a dashboard or a web-page this can be the difference between snappy and unusable.Scale Out Postgres Easily Across Multiple Nodes For Petabyte-Scale DatasetsA common criticism of Postgres is that once you max out your database instance you can’t scale-out effectively. This was why we developed TimescaleDB Multi-node, a way of linking multiple PostgreSQL nodes to scale out ingest and query performance for petabyte-scale datasets while inserting over 1 million rows per second.Using multi-node to provide distributed hypertables automatically spreads data across nodes, still allowing you to access that data as you normally would.Lower Storage CostsThe number one driver of cost for modern time-series applications is storage. Even when storage is cheap, time-series data piles up quickly. Timescale provides two methods to reduce the amount of data being stored, compression and downsampling using continuous aggregates.90 % or More Storage Savings Via Best-In-Class Compression AlgorithmsThe Timescale hypertable is data heavily partitioned into many, many smaller partitions called “chunks.” Timescale providesnative columnar compressionon this per-chunk basis.As we show in the benchmark results (and as we see often in production databases), compression reduced disk consumption by over 90% compared to the same data in vanilla PostgreSQL. Even better, TimescaleDB doesn't change anything about the PostgreSQL storage system to achieve this level of compression. Instead, TimescaleDB utilizes PostgreSQL storage features, namely TOAST, to transition historical data from row-store to column-store, a key component for querying long-term aggregates over individual columns.To demonstrate the effectiveness of compression here’s a comparison of the total size of the cpu table and indexes in TimescaleDB and in PostgreSQL.With the propercompression policy in place, hypertable chunks will be compressed automatically once all data in the chunk has aged beyond the specified time interval. In practice, this means that a hypertable can store data as row-oriented for newer data and column-oriented for older data simultaneously. Having the data stored as both row and column store also matches the typical query patterns of time-series applications to help improve overall query performance—again, something we see in the benchmark results.This reduces the storage footprint and improves query performance even further for many time-series aggregate queries. Compression is also automatic: users set a compression horizon, and then data is automatically compressed as it ages.This also means that users can save significant costs using cloud services that provide separation of compute and storage – such as Timescale – so that larger machines aren’t needed just for more storage.More Storage Savings by Easily Removing or Downsampling DataWith TimescaleDB, automateddata retentionis achieved withone SQL command:SELECT add_retention_policy('cpu', INTERVAL '7 days');There's no further setup or extra extensions to install or configure. Each day any partitions older than 7 days will be dropped automatically. If you were to implement this in vanilla PostgreSQL you’d need to use DELETE to remove records, which is a very costly operation as it needs to scan for the data to remove. Even if you were using PostgreSQL declarative partitioning you’d still need to automate the process yourself, wasting precious developer time, adding additional requirements, and implementing bespoke code that needs to be supported moving forward.One can also combine continuous aggregates and data retention policies to downsample data, and then drop the raw measurements, thus saving even more data storage.Using this architecture, you can retain higher-level rollup values for a longer period of time, even after the raw data has been dropped from the database. This allows multiple different levels of granularity to be stored in the database, and provides even more ways to control storage costs.More Features to Speed Up Development TimeTimescaleDB includes more features that speed up development time. This includes a library of over 100 hyperfunctions, which make complex time-series analysis easy using SQL, such as count approximations, statistical aggregates, and more. TimescaleDB also includes a built-in, multi-purpose job scheduling engine for setting up automated workflows.Library of Over 100 Hyperfunctions That Make Complex Analysis EasyTimescale hyperfunctions make data analysis in SQL easy. This library includes:time-weighted averages,last observation carried forward, anddownsampling with LTTP or ASAP algorithms,time_bucket(), andtime_bucket_gapfill().As an example, one could get the average temperature every day for each device over the last seven days, carrying forward the last value for missing readings with the following SQL.SELECT
  time_bucket_gapfill('1 day', time) AS day,
  device_id,
  avg(temperature) AS value,
  locf(avg(temperature))
FROM metrics
WHERE time > now () - INTERVAL '1 week'
GROUP BY day, device_id
ORDER BY day;For more information on the extensive list of hyperfunctions in TimescaleDB, please visit ourAPI documentation.Built-In Job Scheduler for Workflow AutomationTimescaleDB provides the ability to schedule the execution of custom stored procedures withuser-defined actions. This feature provides access to the same job scheduler that TimescaleDB uses to run all of the native automation jobs for compression, continuous aggregates, data retention, and more.This provides the similar functionality as a third party scheduler likepg_cron, without needing to maintain multiple PostgreSQL extensions or databases.We see users doing all sorts of neat stuff with user-defined actions, from calculating complex SLAs, to sending event emails based on data correctness, to polling tables.Still 100 % PostgreSQL and SQLNotably, because TimescaleDB is packaged as a PostgreSQL extension, it achieves these results without forking or breaking PostgreSQL.Extending PostgreSQL—Not Forking or CloningPostgres is popular at the moment, but a lot of that popularity is with ‘Postgres compatible’ products which might look like Postgres, or talk like Postgres, or query somewhat like Postgres - but aren’t Postgres under the hood (and are sometimes closed-source).TimescaleDB is just PostgreSQL. One can install other extensions, make full use of the type system, and benefit from the incredibly diverse Postgres ecosystem.100 % SQLAny product which can connect to PostgreSQL can query time-series data stored with TimescaleDB using the same SQL it normally would. While we provide helper functions for working with data, we do not restrict the SQL features one can use. Once in the database, users can combine time-series and business data as necessary.Rock Solid Foundations Thanks to PostgreSQLPostgreSQL is not a new database: it has years of production deployments under its belt. High availability, backup and restore, and load-balancing are all solved problems. As we mentioned earlier, we chose Postgres because it was reliable, and TimescaleDB inherits that reliability.Benchmarking Setup and ResultsThis section provides details about how we tested TimescaleDB against vanilla PostgreSQL. Feel free to download theTime-Series Benchmarking Suiteand run it for yourself. If you'd like to get started with TimescaleDB quickly you can use Timescale, which lets yousign up for a free, 30-day trial.Benchmark ConfigurationFor this benchmark, all tests were run on the same m5.2xlarge EC2 instance in AWS us-east-1 with the following configuration and software versions.Versions: TimescaleDB version 2.7.2, community edition, and PostgreSQL 14.4One remote client machine running TSBS, one database server, both in the same cloud datacenterTSBS Client Instance: EC2 m5.4xlarge  with 16 vCPU and 64 GB memoryDatabase server instance: EC2 m5.2xlarge  with 8 vCPU and 32 GB memoryOS: both server and client machines ran Ubuntu 20.04Disk size: 1 TB of EBS GP2 storageTSBS config: Dev-ops profile, 4,000 devices recording metrics every 10 seconds over one month.We also deliberately chose to use EBS (elastic block storage) volumes rather than attached SSDs. While benchmark performance would certainly improve with SSDs, the baseline performance using EBS is illustrative of what many self-hosted users could expect while saving some expenses by using elastic storage.Database ConfigurationWe ran only one PostgreSQL cluster on the EC2 database instance. The TimescaleDB extension was loaded viashared_preload_librariesbut not installed into the PostgreSQL-only database.To set sane defaults for the PostgreSQL cluster, we rantimescaledb-tuneand setsynchronous_commit=offin postgresql.conf. This is a common performance configuration for write-heavy workloads while still maintaining transactional, logged integrity.All configuration changes applied to both PostgreSQL and TimescaleDB benchmarks alike.The DatasetAs we mentioned earlier, for this benchmark, we used theTime-Series Benchmarking Suiteand generated data for 4,000 devices, recording metrics every 10 seconds, for one month. This generated just over one billion rows of data. Because TimescaleDB is a PostgreSQL extension, we could use the same data file and ingestion process, ensuring identical data in each database.TimescaleDB SetupTimescaleDB uses an abstraction called hypertables which splits large tables into smaller chunks, increasing performance and greatly easing management of large amounts of time-series data.We also enabled native compression on TimescaleDB. We compressed everything but the most recent chunk of data, leaving it uncompressed. This configuration is a commonly recommended one where raw, uncompressed data is kept for recent time periods and older data is compressed, enabling greater query efficiency. The parameters we used to enable compression are as follows: we segmented by thetags_idcolumns and ordered by time descending andusage_usercolumns.All benchmark results were performed on a single PostgreSQL table and on an empty TimescaleDB hypertable created with four-hour chunks.(And for those thinking that we also need to compare TimescaleDB with PostgreSQL Declarative Partitioning: please read on to the end, we discuss that as well.)Query Latency Deep DiveFor this benchmark, we inserted one billion rows of data and then ran a set of queries 100 times each against the respective database. The data, indexes, and queries are exactly the same for both databases. The only difference is that the TimescaleDB queries use thetime_bucket()function for doing arbitrary interval bucketing, whereas the PostgreSQL queries use the newdate_bin()function, introduced in PostgreSQL 13.Query latency comparison between PostgreSQL and TimescaleDB, for different queriesThe results are clear and consistently reproducible. For one billion rows of data spanning one month of time (with four-hour partitions),TimescaleDB consistently outperformed a vanilla PostgreSQL database running 100 queries at a time.There are two main reasons for Timescale's consistent query performance.Compression = Smaller Storage + Less WorkIn PostgreSQL (and many other databases), table data is stored in an 8 Kb page (sometimes called a block). If a query has to read 1,000 pages to satisfy it, it reads ~8 Mb of data. If some of that data had to be retrieved from disk, then the query will usually be slower than if all of the data was found in memory (the reserved space known asshared buffersin PostgreSQL, if you’re looking for some insight into PostgreSQL caching we havea blog on that).With TimescaleDB compression, queries that return the same results have to read significantly fewer pages of data (this is both because of the actual compression and because it can return single columns rather than whole rows). For all of our benchmarking queries, this also translates into higher concurrency for the benchmark duration.Stated another way, compression typically impacts fetching historical data most because TimescaleDB can query individual columns rather than entire rows. Because less I/O is occurring for each query, TimescaleDB can handle more queries with a lower standard deviation than vanilla PostgreSQL.Let's look at two examples of how this plays out between the two databases using two queries above,cpu-max-all-1andsingle-groupby-1-1-12.single-groupby-1-1-12We selected one of the queries from the benchmark and ran it on both databases. Recall that each database has the exact same data and indexes on uncompressed data. TimescaleDB has the advantage of being able to segment and order compressed data in a way that's beneficial to typical application queries.EXPLAIN (ANALYZE,BUFFERS)
SELECT time_bucket('1 minute', time) AS minute,
        max(usage_user) as max_usage_user
        FROM cpu
        WHERE tags_id IN (
          SELECT id FROM tags WHERE hostname IN ('host_249')
        ) 
        AND time >= '2022-08-03 06:16:22.646325 +0000' 
        AND time < '2022-08-03 18:16:22.646325 +0000'
        GROUP BY minute ORDER BY minute;When we run theEXPLAINon this query and ask forBUFFERSto be returned, we start to get a hint of what's happening.Query latency vs volume of data that has to be read to satisfy the query in TimescaleDB and PostgreSQL.Two things quickly jump out when I view these results. First, the execution times are significantly lower than the benchmarking results above. Individually, these queries execute pretty fast, but PostgreSQL has to read approximately 27x more data to satisfy the query. When 16 workers request data across the time range, PostgreSQL has to do a lot more I/O, which consumes resources. TimescaleDB can simply handle a higher concurrency for the same workload. We can see this in the full benchmarking output, too.cpu-max-all-1Again we can clearly see the impact of compression on the ability for TimescaleDB to handle a higher concurrent load when compared to vanilla PostgreSQL for time-series queries.EXPLAIN (ANALYZE, buffers) 
SELECT
   time_bucket('3600 seconds', time) AS hour,
   max(usage_user) AS max_usage_user,
   max(usage_system) AS max_usage_system,
   max(usage_idle) AS max_usage_idle,
   max(usage_nice) AS max_usage_nice,
   max(usage_iowait) AS max_usage_iowait,
   max(usage_irq) AS max_usage_irq,
   max(usage_softirq) AS max_usage_softirq,
   max(usage_steal) AS max_usage_steal,
   max(usage_guest) AS max_usage_guest,
   max(usage_guest_nice) AS max_usage_guest_nice 
FROM cpu 
WHERE  
   tags_id IN (
      SELECT id FROM tags WHERE hostname IN ('host_249')
   )
   AND time >= '2022-08-08 18:16:22.646325 +0000' 
   AND time < '2022-08-09 02:16:22.646325 +0000' 
GROUP BY HOUR 
ORDER BY HOUR;Query latency vs volume of data that has to be read to satisfy the query, in TimescaleDB and PostgreSQLWith compression, TimescaleDB does significantly less work to retrieve the same data, resulting in faster queries and higher query concurrency.Time-Ordered Queries Just Work BetterTimescaleDB hypertables require a time column to partition the data. Because time is an essential (and known) part of each row and chunk, TimescaleDB can intelligently improve how the query is planned and executed to take advantage of the time component of the data.For example, let's query for the maximum CPU usage for each minute for the last 10 minutes.EXPLAIN (ANALYZE,BUFFERS)        
SELECT time_bucket('1 minute', time) AS minute, 
  max(usage_user) 
FROM cpu 
WHERE time > '2022-08-14 07:12:17.568901 +0000' 
GROUP BY minute 
ORDER BY minute DESC 
LIMIT 10;Because TimescaleDB understands that this query is aggregating on time and the result is ordered by the time column (something each chunk is already ordering by in an index), it can use the ChunkAppend custom execution node. In contrast, PostgreSQL plans five workers to scan all partitions before sorting the results and finally doing aGroupAggregateon the time column.Query latency vs volume of data that has to be read to satisfy the query, in TimescaleDB and PostgreSQLTimescaleDB scans fewer data and doesn't need to spend time re-sorting the data that it knows is already sorted in the chunk. For time-series data with a known order and constraints, TimescaleDB works better for most queries than vanilla PostgreSQL.Ingest PerformanceIntriguingly, ingest performance for both TimescaleDB and PostgreSQL are nearly identical, a dramatic improvement for PostgreSQL given theresults five years ago with PostgreSQL 9.6. However, TimscaleDB still consistently finished with an average rate of 3,000 to 4,000 rows/second higher than a single PostgreSQL table.Insert performance comparison between TimescaleDB 2.7.2 and PostgreSQL 14.4This shows that while vast improvements have been made in PostgreSQL, Timescale hypertables also continue to perform exceptionally well. As well as the rate, the other characteristics of ingest performance are nearly identical between TimescaleDB and PostgreSQL. Modifying the batch size for the number of rows to insert at a time impacts each database the same: small batch sizes or a few hundred rows significantly hinder ingest performance, while batch sizes of 10,000 to 15,000 rows seem to be about optimal for this dataset.Declarative PartitioningIn the benchmarks above we tested TimescaleDB against a single PostgreSQL table, simply because that’s the default option that most people end up using. PostgreSQL also has support for native declarative partitioning, which has also been maturing over the past few years.For the sake of completeness, we also tested TimescaleDB against native declarative partitioning. As the graphic below shows TimescaleDB is still 1000x faster for some queries, with strong performance gains still showing across the board. Ingest performance was similar between TimescaleDB and declarative partitioning.In fact, if anything, the takeaway from these tests was that while declarative partitioning has matured, the gap between using a single-table and declarative partitioning has shrunk.Query latency comparison between TimescaleDB and PostgreSQL with declarative partitioningUsing declarative partitioning is also harder. One needs to manually pre-create partitions, ensure there are no data gaps, ensure no data is inserted outside of your partition ranges, and create more partitions as time moves on.In contrast, with TimescaleDB, one does not need any of this. Instead, a singlecreate_hypertablecommand is used to convert a standard table into a hypertable, and TimescaleDB takes care of the rest.ConclusionTimescaleDB harnesses the power of the extension framework to supercharge PostgreSQL for time-series and analytical applications. With additional features like compression and continuous aggregates, TimescaleDB provides not only the most performant way of using time-series data in PostgreSQL, but also the best developer experience.When compared to traditional PostgreSQL, TimescaleDB enables 1,000x faster time-series queries, compresses data by 90 %, and provides access to advanced time-series analysis tools and operational features specifically designed to ease data management. TimescaleDB also provides benefits for other types of queries with features like SkipScan—just by installing the extension.In short, TimescaleDB extends PostgreSQL to enable developers to continue to use the database they love for time series, perform better at scale, spend less, and stream data analysis and operations.If you’re looking to expand your database scalability, try our hosted service,Timescale. You will get the PostgreSQL you know and love with extra features for time series (continuous aggregation,compression,automatic retention policies,hyperfunctions). Plus, a platform withautomated backups, high availability, automatic upgrades, flexible resizing with autoscaling, and much more.You can use it for free for 30 days; no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/postgresql-timescaledb-1000x-faster-queries-90-data-compression-and-much-more/
2023-08-23T11:00:00.000Z,What Is Time-Series Data? Definitions & Examples,"What Is Time-Series Data?Time-series data is a sequence of data points collected over time intervals, allowing us totrack changes over time. Time-series data can track changes over milliseconds, days, or even years.Having access to detailed, feature-rich time-series data has become one of the most valuable commodities in our information-hungry world. Businesses, governments, schools, and communities, large and small, are finding invaluable ways to mine value from analyzing time-series data.(You can read how real-world teams, like those trackingreal-time flight dataor building platforms forsmarter cities, analyze their time-series metrics in ourDeveloper Q&A series).Software developer usage patterns already reflect the same trend. In fact, over the past two years, time-series databases (TSDBs, or time-series DBMS—database management systems) have steadily remained the fastest-growing category of databases:Source:DB-Engines, August 2022As the developers of anopen-source time-series database, my team and I are often asked about this trend and how it should factor into your decisions about which database to select. Specifically, does it really matter if you start with adatabase specialized for time-series data—or can you easily transition to one later?Why Is Time-Series Data Valuable?Once you start capturing time-series data, a brand new world of analytics and insight opens up to you. On the flip side, recording data points this way produces much more data overall. For example, think about the stock market. Technically, you could (andpeople do!) record the price changes of a single stock a hundred times every second.For any given day, that is 24 hours x 60 minutes x 60 seconds x 100 = 24*60*60*100 = 8,640,000 data records each day. And this is just one single stock symbol. This calculation ignores the time window when the market is closed every day but still shows that time-series data can be high-volume, even if you just monitor a single stock symbol.What Are Some Examples of Time-Series Data?In the past, our view of time-series data was more static; the daily highs and lows in temperature, the opening and closing value of the stock market, or even the daily or cumulative hospitalizations due to COVID-19.Unfortunately, these totals missed the nuances of how the underlying changes over time contributed to these static values.Let’s consider a few examples.❓Do you know if you have time-series data?Unveiling banking nuances with time-series databasesIf I send you $10, a traditional bank database will debit my account and credit your account. Then, if you send me $10, the same process happens in reverse. At the end of this process, our bank balances would look the same, so the bank might think, “Oh, nothing changed this month.” But, with a time-series database, the bank would see, “Hey, these two people keep sending each other $10; there’s likely a deeper relationship here.” Tracking this nuance, our month-ending account balance takes on greater meaning.Tracking environmental efficiency with time seriesNext, think about an environmental value like mean daily temperature (MDT), the average of the high and low temperature for consecutive days at a location. Over the last few decades, MDT has been used as a primary variable to calculate buildings’ energy efficiency.In any given week, MDT might only vary slightly from day to day in a location, but the contributing environmental factors could be changing drastically over that same period. Instead, knowing how the temperature changed each hour throughout the day, coupled with precipitation, cloud cover, and wind speed, could dramatically improve your ability to model and optimize energy efficiency for your properties.Time-series analysis of COVID-19 hospitalizationsLikewise, while knowing the total number of COVID-19 hospitalizations per day in your community is valuable, that numberaloneisn’t very descriptive. For instance, the hospital might disclose daily numbers that show 20 hospitalizations on Monday and increase slightly throughout the week to 23 hospitalizations on Friday.At first glance, it looks like a 15 % increase in hospitalizations this week. But, if we add detail to each of those records (and increase the frequency at which we collect them), we might see that it was anetincrease of three patients. In reality, there were 10 people discharged, and 13 admitted, an increase of 65 % for new admissions over the last five days.Tracking each aspect of patient data over time (e.g., patient age, admitted or discharged, days to recovery, etc.) helps us understand how we arrive at the daily counts, allowing us to better analyze trends, accurately report totals, and take action. In the case of total COVID-19 hospitalizations, the details behind this analysis impact public policy in the cities and towns where we live.Financial marketsThe financial sector is a typical example of time-series data usage: be it stocks, cryptocurrencies, or other financial assets, time-series data allows you to see how prices changed over time and helps you spot trends. As an example, here’s a time-series chart showing you the intraday price changes of the Bitcoin cryptocurrency:Time-series data allows you not just to know the current price of the asset but also how it changed in the past.Internet of Things and sensor dataWhether you’re recordingmotor temperatures in factories, monitoringcannabis cultivation, or even using IoT data to control anuclear fusion experiment, you are leveraging time-series data to make better decisions.Once you have sensors that send data into your time-series database, you can create real-time dashboards and analyze historical data.Application monitoringImagine you maintain a web application. Every time a user logs in, you may just update alast_logintimestamp for that user in a single row in youruserstable. But what if you treated each login as a separate event and collected them over time? With that kind of time-series data, you could analyze historical login activity, see how usage increases or decreases over time, bucket users by how often they access the app, and more.ObservabilityAnother example that has become vital to every IT group around the world: operational metrics for servers, networks, applications, environments, and more. This kind of time-series metric data is crucial to keeping the services we rely on running without interruption. By tracking the changes in each metric, IT departments can quickly identify problems, plan for capacity increases during upcoming events, and diagnose if an application update resulted in changed user behavior, for better or worse. (See how NLP Cloud monitors their language AI API.)Web3 and blockchain dataIn the past year, we’ve seen a surge in companies that use TimescaleDB to buildweb3 and blockchain tools. Blockchains are made of timestamped blocks and transactions. There are several types of data to be recorded to drive smarter decisions in the industry. Think of NFT transaction monitoring, blockchain exploration, mining analytics, or even criminal investigations.These examples illustrate how modern time-series data differs from what we’ve known in the past. Time-series data analysis goes far deeper than a pie chart or Excel workbook with columns of summarized totals.This detailed data doesn’t just include time as a metric but as a primary component that helps to analyze our data and derive meaningful insights.And, there are many other kinds of time-series data. Still, regardless of the scenario or use case, all time-series datasets have three things in common:The data that arrives is almost always recorded as a new entry.The data typically arrives in time order.Time is a primary axis (time intervals can be either regular or irregular).In other words, time-series data workloads are generally “append-only.” While they may need to correct erroneous data after the fact or handle delayed or out-of-order data, these are exceptions, not the norm.Simply put, time-series datasets track changes to the overall system as INSERTs, not UPDATEs, resulting in an append-only ingestion pattern.This practice of recording each and every change to the system as a new, different row is what makes time-series data so powerful.It allows us to measure and analyze change: what has changed in the past, what is changing in the present, and what changes we forecast may look like in the future.You may also notice that some of these examples describe a common type of time-series data known asevent data.Now, It’s Your Turn: Start Analyzing Your Time-Series DataIf you’re convinced you need a time-series database or want to try it out,spin up a fully managed TimescaleDB instance—free for 30 days.From there,follow our getting started guideto configure your database and execute your first query, then choose one of our fun tutorials to delve deeper into TimescaleDB:Introduction to time-series forecastingGetting Started with Grafana and TimescaleDBMore tutorials!You can also read more primer content on time-series data and what you can do with it:What Is a Time-Series Database, and Why Do I Need One?Best Time-Series Database: How to ChooseWhat Is Time-Series Analysis? (With Examples and Applications)What Is Time-Series Forecasting?What Is a Time-Series Graph With ExamplesWhat Is a Time-Series Plot, and How Can You Create OneHave questions or want to learn more? Join ourSlack CommunityandForum, where Timescale engineers and community members are active in all channels.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/time-series-data/
2022-12-22T15:30:33.000Z,Year of the Tiger: 12 (and Then Some) Timescale Highlights of 2022,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Twenty twenty-two. What a year, huh?As the old year comes to an end and a new year approaches, it’s time to look back at all that we accomplished in 2022 and start setting goals and making wishes for 2023.Every holiday season, we resume the Timescale tradition of sharing our achievements and favorite content pieces to celebrate#12DaysofTimescale—the kind of retrospective all developers can get behind.This year, though, we’re adding a little twist. Keep reading for a recap of our major launches (with helpful links for your time-series work), team achievements, and the highlights of the year—including bits from an infamous, mic-dropping retrospective by our CTO and my fellow co-founder,Mike Freedman.Join us as we remember this fun ride!We Closed Our Series C 🦄 Funding RoundWell, 2022 was the Year of the Tiger 🐯, and who are we to deny it? We serendipitously kicked off 2022 with theannouncement that we had raised $110 million in our Series C, led by Tiger Global (seriously, westilldon’t believe in coincidences).With the funding, Timescale’s valuation surpassed $1 billion. Combined with previous rounds (2018,2019,2021), we raised more than $180 million to drive growth and enable developers to build the next wave of computing. And with great power comes great responsibility, so not only are we empowering developers, we have created lots of software and new features to make their jobs easier.We’re also pretty good at celebrating our accomplishments. Case in point? Just check out our social media marketing manager,Jill Proman, celebrating the news of our Series C in Times Square, New York.We Launched Some Cool Database Features 🚀TimescaleDB 2.6: Introducing compression for continuous aggregatesWe believe our mission is to help developers, first and foremost.So, when we heardusers requesting compression for their continuous aggregates, we couldn’t say no. Continuous aggregates,our improved version of PostgreSQL materialized views, already allowed developers to downsample their time-series data considerably since they could keep their continuous aggregates even when the original raw data had been dropped from the underlying hypertable.But guess what? Timescale users are handling huge volumes of time-series data—our community wanted to use compression over their continuous aggregates to save even more disk space. This functionality was introduced in TimescaleDB 2.6, allowing users to combine continuous aggregates, compression, and data retention policies to save up to +95 % in storage.Support for compression in continuous aggregates was a highly requested feature by our community, which we introduced in TimescaleDB 2.6TimescaleDB 2.7: Making data aggregation faster and betterBy popular demand, continuous aggregates were also the protagonists of TimescaleDB 2.7. It’s no surprise that they’re among Timescale’s most popular features—continuous aggregates are aimed at solving one of the most difficult challenges for developers and data scientists working with time-series data:aggregating it efficiently without having to query billions or trillions of raw data rows.If with Timescale 2.6 we helped users compress their continuous aggregates to achieve incredible savings, in Timescale 2.7, we introduced a completely redesigned materialization process that made them better and faster, dramatically boosting performance.First, continuous aggregates became blazing-fast—and by blazing-fast, we mean up to 44,000x faster in some queries than previous versions. Second, they became smaller: in TimescaleDB 2.7, continuous aggregates needed 60% less storage on average (which directly translates into more cost savings!). Lastly, we also removed limitations around continuous aggregates, like the possibility of usingDISTINCT,FILTER, orORDER BY.Some queries became blazing-fast with TimescaleDB 2.7. For example, we saw nearly 45,000x better performance on ORDER BYs in queries searching only through materialized dataBut that’s not all we did with Timescale 2.7. Once again, we decided to cater to our community by solving an issue that often popped up in ourCommunity Slack,Support, andForum(the latter is also a 2022 highlight, but we’ll get there).The problem revolved around thehigh planning time in the presence of many chunks (data partitions within a database table) in queries using thenow ()function, which had been haunting us in partitioned vanilla PostgreSQL too. (Hey, needless to say, we’re heavy PostgreSQL users and fans!)Long story short, not only did we optimize the function, but we made it scale with the number of chunks in hypertables. So, the more data partitions you’re dealing with, the more you’ll notice the speed improvement—up to 401x faster for 20,000 chunks compared to the previous version.Not bad for the first six months of the year. 🐯TimescaleDB 2.8: Time zones intime_bucketWhen we inquired our team about their 2022 highlights, developer advocateChris Engelbertwas the first to jump up and reply: the updates to thetime_buckethyperfunction that we shipped with TimescaleDB 2.8!time_buckethas always been a superstar among the Timescale features. It allows developers to easily group their time series by arbitrary buckets of time, for example, by day or week (like an improved version of PostgreSQL’sdate_trunc). But before TimescaleDB 2.8, there was some highly-requested functionality missing intime_bucket: the possibility of bucketing by month and year and specifying time zones in the interval definitions.Before becoming a Timescale developer advocate, Chris was a Timescale user in his own startup andhad nightmares about manually defining his customers’ time zone on his views. It’s no wonder he was over the moon with TimescaleDB 2.8, which introduced this out-of-the-box fortime_bucket, enabling super time-based queries and reporting by supporting bucketing by month, year, and time zone.Since TimescaleDB 2.8, you can now specify timezones in time_bucket, simplifying your views' definition if you have customers in multiple time zonesTimescaleDB 2.9: Continuous aggregates on continuous aggregatesWe already mentioned that continuous aggregates are one of our most popular features, so why not have continuous aggregates on top of other continuous aggregates?For our final TimescaleDB release of the year(which will be available in Timescale and Managed Service for TimescaleDB in January 2023), we wanted to give users the possibility of creating continuous aggregate hierarchies.Each continuous aggregate is defined on top of the previous one, providing a lower granularity view of the same dataset but making the refresh process for higher-level continuous aggregates lightning-fast.⚡This also enables the developer to define more diverse sets of continuous aggregates without significant additional costs.For example, a continuous aggregate that provides a daily summary only needs to go through 24 records when built on top of a continuous aggregate that provides an hourly summary—forget those tens of thousands of records stored in the raw hypertable for that day.Diagram example of the new continuous aggregates functionality for a finance use caseWe also continued to improve thetime_bucket_gapfillfunction to allow specifying the timezone to bucket, and introduced fixed schedules for background jobs, the ability to check job errors, and the option of configuring the availability of the data node, among other noteworthy features.We Reduced Downtime for our End Users With One-Click Replicas 👯Mikeand I are constantly bouncing ideas off each other. Oh! Mike is our co-founder and CTO.For the most part, we're always thinking about how to make our customers' lives easier. This formed a huge piece of 2022—simplifying operational database management for our customers.If you’re working with production databases, a big part of this peace of mind is knowing that your end users won’t experience any significant downtime in case anything happens to your database—and that you won’t be called in the middle of the night to put things up again. This is where high availability replicas come in,a feature we launched earlier this year in Timescale.Database replication in Timescale is as easy as pressing a button. If your database fails, your connections will be automatically switched to the replica, and your end users will only experience a few seconds of downtime. But replicas in Timescale are not only useful for high availability: they can also improve your performance, as they also act as read replicas—you can direct your heavy read queries to the replica, freeing up resources in your main database for writes.If anything happens to your primary database, Timescale will automatically promote the replica to the primary role, and your end users will only experience a few seconds of downtimeIf you want to learn more about how this works under the hood,read this blog post to learn how high availability works in Timescale.We Allowed Developers to Do Faster and Safer Testing With One-Click ForkingForking is another way we made our customers’ workflows easier in Timescale.Again, we made it effortless: with a click of a button, you can spin up forks, identical copies of your database but—and this is a very convenient “but”—with different resource configurations.You can use forks to quickly spin up databases with production data on them, deleting them also with a click of a button once you’re done with them (and you’ll only be charged for the time the fork has been running).This feature can make your life easier in so many ways:You can use forks to run tests in up-to-date production data.You can use forks to create and refresh staging environments.You can use forks to test upgrades.You can use forks to provide access to production data but not to the production databases and much more.“You can spin up another service for another team member for testing and actually spin up a different size,” summarized Mike at our latest All Hands, stressing the flexibility and time savings the feature brings development teams.In Timescale, creating copies of your database is as easy as clicking on “Fork service.” You can delete the fork just as quickly once you’re done with itThanks to You, We Learned More About the PostgreSQL Community With the 2022 State of PostgreSQL SurveyAfter a pandemic hiatus in 2020, ourState of PostgreSQLsurvey was back for its third edition, advancing our desire to provide greater insights into the vibrant and growing PostgreSQL user base.In 2022, nearly one thousand developers answered the survey, more than double compared of the previous year.Download the full reportto learn more about this amazing community, orread this blog post if you want to know what PostgreSQL committer and developer, Heikki Linnakangas, had to say about the survey results as we celebrated PostgreSQL’s 25th birthday!We Have Grown the Timescale Team (and Met in Person!)Without growing our team, we couldn’t make database developers’ lives easier and release new features to allow them to build the next wave of computing. Such ambitious goals require efficiency and constraint but also the right talent to solve complex problems.The Observability team during their first offsite in Valencia, Spain, in AprilThe past 12 months have been huge for the Timescale Team: in 2022, we nearly doubled in size, hiring one hundred new Timescalers, half of which for the Engineering and Product teams. No wonder recruiting manager Shauna Lassche considers it one of 2022’s highlights: “We hired some awesome people who are making an impact on our product.”For those on the other end of the recruiting spectrum, joining a global and diverse company with a fun-loving culture was also a highlight. At least that’s what senior product manager Jim Meehan promptly replied in our highlight Slack appeal, having just joined just a few weeks ago: “It’s so exciting to be part of an employee-first company. It’s my first week, and I’ve been wowed by the onboarding experience and overall company culture.”Alaap Murali, another senior product manager who responded to our Slack call, said: “I’m so grateful to work with so many wonderful and talented people!” So if you want to join our fully remote, global, and diverse team,check out our careers pageand help us build the next great database company.Mike and software engineer James Guthrie enjoying breakfast in Switzerland in JulyBut even more significant than working remotely together, one of the most wonderful things of 2022 was the team’s numerous face-to-face meetings, whether at Timescale offsites, spontaneous team gatherings, or events, such as AWS Re:Invent. It always warms our hearts when Timescalers finally meet in person, and this year we could finally catch up following the quiet pandemic years.We Did Some Benchmarking, Not BenchmarketingWhen you live and breathe data, making informed data-driven decisions matters. This is why we love benchmarking our products against others, backing up our claims with rigorous numbers and facts. There is a line we always say at Timescale: we do benchmarks, not benchmarketing.In September,we showed how TimescaleDB expands PostgreSQLfor time series and analytics by improving query performance by up to 1,000x or more, reducing storage by 90 %, and providing additional crucial features for time-series applications and analytics.TimescaleDB expands PostgreSQL with faster queries, storage savings, and time-saving features for time series and analyticsA few months later, in November,we answered why so many developers are migrating from Amazon RDS for PostgreSQL to Timescale: in our head-to-head challenge, Timescale delivered up to 350x faster queries, a speedier ingest by 44 %, and storage savings of up to 95 % for time-series data.If you’re running on RDS, Timescale can give you more ingest and faster queries and help you save money in storageWe Turned Promscale Into a Full Observability BackendThis year, the Promscale Team has made incredible progress in developing a scalable metric and trace observability backend for Prometheus, Jaeger, and OpenTelemetry.The main idea behind Promscale is, again, to simplify developer workflows (so they can focus on what really matters) by offeringobservability powered by SQL. With PostgreSQL and TimescaleDB, we provide you with a robust and scalable database for time-series data. But what about using all the data from your complex distributed systems and cloud-native architectures to monitor their performance and prevent dysfunctional behavior?By giving developers support for tracing and alerts and allowing them to add and store their OpenTelemetry data, Promscale bridges that gap. Devs can now understand how their systems’ components work and interact with each other in real-time.Then, in October,Promscale became one of the only two external certified storage backends for Jaeger, a popular tool for distributed tracing (if you want to learn more about this,check out this blog post we recently wrote).In addition to passing all the Jaeger storage certification tests, Promscale ensures a rich query experience with SQL since it’s built on PostgreSQL and TimescaleDB. Gaining in-depth insights into your systems was never this easy—and that is definitely something to celebrate.We Had (Literally) an Eventful YearWhile we love meeting each other, another huge highlight of 2022 was getting ourselves out there and meeting many of you face-to-face at conferences and meetups. Thank you so much to all of you who dropped by our booths!This year, we attended or sponsored43 events(that’s almost four events per month), from PostgreSQL conferences on both sides of the Atlantic (PG Conf NYCandPostgreSQL Conference Europe) to the massiveAWS Re: Invent, in Las Vegas, in November.Showing our demos in person, listening to your feedback, and finding ways to solve your use cases was undoubtedly one of the year's highlights.The Timescale Team at AWS Re:Invent in Las Vegas, last NovemberWe Launched S3 Data Tiering for TimescaleBarely a month ago, we announced a very exciting private beta:we’re building a consumption-based, bottomless object store in Timescale.This functionality expands the limits of traditional cloud PostgreSQL databases by allowing you to store the data in your relational table both in our traditional storage (built on standard disk-based storage EBS) and in the object storage layer (built on Amazon S3).You’ll keep the illusion of a single relational table, interacting with it via standard SQL just as you’re used to doing it with PostgreSQL. But under the hood, you have access to infinite, low-cost scalability, paying only for what you store.Storing data in the object store is as easy as a single SQL command to automatically tier data based on its age, as suited to your application’s needs:# Store data older than two weeks in the object storage layer
SELECT add_data_tiering_policy(‘metrics’, INTERVAL ‘2 weeks’);And this is only the beginning: we have big plans. We’ll further improve object storage in Timescale to not only serve as bottomless, cost-efficient storage but also as ashared storage layerthat will make it dramatically easier for developers to share data across their entire fleet of databases.We’re inspired by our vision to build a data infrastructure that extends beyond the boundaries of a traditional database, combining the flexibility of a serverless platform with all the performance, stability, and transparency of PostgreSQL that developers know and love.This functionality is available for testing in private beta for all Timescale users.Sign up for Timescaleand navigate to the Operations screen, pictured below, to request access. Timescale is free for 30 days, no credit card required.You can access data tiering to S3 from the Timescale UI. Just click on “Request Access,” and we’ll be in touch soon with the next stepsWe Built a Better Developer Experience With Timescale 2.0It took us about a year to complete and involved a massive company-wide effort between designers, engineers, and product designers who spent a lot of time planning, building, and iterating the new user interface of our cloud-native database platform. But it was all worth it.We presented Timescale 2.0 earlier this year, in December,and it is one of 2022’s biggest wins, both for the team who laboriously redesigned it but also for every Timescale developer who now enjoys a seamless developer experience.I highly recommendchecking out the blog post the team wroteabout this redesign journey and browsingtheir interactive prototypeof the UI library.All Thanks to You!Yes, you (reading this)! Whether this is your first engagement with us or you’ve been here through the year, you’re a highlight in our story. As mentioned, Timescale is a help-first company: our goal is to help developers solve problems, help move the industry forward technically, and make it more welcoming and diverse to all. We want to enable developers to build the next wave of computing, and we couldn’t do it without you and your feedback.This is why we’re constantly focused on maintaining and opening new communication channels with our users, such as ourSlack Community(with nearly 10,000 users!) and theTimescale Forum, which we kicked off in January, right at the start of 2022.And Then SomeIf this sounds like a lot, brace yourself—there is more to come. We weren’t kidding when we said that 2022 was ahugeyear for us. In this Year of Tiger, our Engineering Team also reinforcedour Kubernetes infrastructure for Timescale.Timescale runs in Kubernetes, and this year, we introduced a custom runtime mode for Patroni, ensuring that our customers’ TimescaleDB clusters will not be negatively impacted byetcd failures. Due to this work, instances on Timescale can continue operating safely and smoothly even in adverse runtime networking conditions.We also introduced a new methodology for installing extension updates without downtime and built tools for detecting and mitigating out-of-memory (OOM) errors before they cause trouble. This made Timescale a highly resilient platform, with zero system-wide outages almost every month.And finally, let’s round this blog post off with a few more numbers (because we ♥️ data):# of GitHub issues closed:367 issues# of TimescaleDB instances created: more than 36 million TimescaleDB instances created over the last 365 days (we’re aware this figure does not reflect actual usage, as some were dropped, but still!)# of releases: 9 releases this year (including patches) for TimescaleDB, 10 releases and patches for Promscale, and 9 main releases and patches for Timescale (these are just the main ones because trust us, we had smaller releases practically every day) 🎉# of PostgreSQL commits: 18 authored, 19 reviewed, and 5 reportedAfter completing all this work for our fantastic community, we’re looking forward to the holiday season for some well-deserved rest and time with our family and friends. But we could not be prouder of what we achieved in 2022, and we are honored to share it with you.We wish you a happy, healthy, and hopeful holiday season! Here’s to an even better 2023! 🎉Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/year-of-the-tiger-12-and-then-some-timescale-highlights-of-2022/
2023-02-10T13:30:38.000Z,Time-Series Database: An Explainer,"People create, capture, and consume more datathan ever before. And a big portion of this data volume is timestamped, a.k.a. time-series data.So it’s not surprising that the time-series database (TSDB) category has seen a lot ofgrowth in popularityin the past five years.In this post, we’ll introduce you to time-series databases and how they can work for you:What is a time-series database?Is my data time-series data?Why businesses depend on time-series databasesPopular time-series databasesHow do you choose one?Comparing top brands: InfluxDB vs Timescale DBFurther resourcesFAQsWhat Is a Time-Series Database?A time series database is a type of database specifically designed for handling time-stamped or time-series data.Time-series data are simply measurements or events that are tracked, monitored, downsampled, and aggregated over time.This could include server metrics, application performance monitoring data, network data, sensor data, and more. Whether you are recording the temperature in your garden, the price of a stock, or monitoring your application’s usage data, you are dealing with time-series data.Since time-series data is time-centric, recent, andnormally append-only, a time-series database (TSDB) leverages these foundational characteristics to store time-series data more simply and efficiently than general databases.Time-series databases vs traditional databasesYou might ask: Why can’t I just use a “normal” (i.e., non-time-series) database?Well, you can, and some people do.But relational databases like PostgreSQL get too slow for time-series data once tables start getting big.Time-series data accumulates very quickly, and relational databases are not designed to handle that scale (at least not in an automated way).Traditionally, relational databases fare poorly with vast datasets, and NoSQL databases are hailed as the best performers at scale.But the truth is that,with some engineering fine tuning, PostgreSQL can be turned into a time-series database—actually performing much better than other specialized solutions (as we've shown in benchmarksversus InfluxDB,Cassandra, andMongoDB).Time-series databases—whether relational or NoSQL-based—introduce efficiencies that are only possible when you treat the time element as a first-class citizen.These efficiencies allow them to offer massive scale, from performance improvements, including higher ingest rates and faster queries at scale (although some support more queries than others) to better data compression.Is My Data Time-Series Data?Even if you don't often refer to your data as time-series data, they might be. A few examples of specific time-series data types:Sensor dataTransaction data / Financial transactions / Customer transactions / Order historyOperational analytics / Application dataFleet data /LogisticsMetrics dataTick data / Fintech data / Trading dataEvent dataVector dataWeather dataInsurance dataCall recordsIf you want to know if your data is time-series data, this is the litmus test: does your data have some kind of timestamp or time element related to it, even if that may not be its main dimension? If the answer is “yes,” you’re dealing with time-series data.Why Businesses Depend on Time-Series DatabasesThere are several reasons for this trend:The increasing number of IoT devices and applications has increased the need for databases that can store time-series data efficiently. IoT often generates high-granularity and high-volume datasets that general-purpose databases can’t handle very well.Most companiesplan to invest morein data and analytics. As organizations seek to gain insights from both their historical and real-time data, the demand for time-series databases has increased.Therise of observabilityand the growing importance of monitoring your application at all times have also contributed to the popularity of time-series databases since this kind of data is also time-series data.How does a time-series database help?A time-series database helps you worry less about your database infrastructure, so you can spend more time building your application or gaining insight from the data.You can make it simpler or faster (or both) to ingest and query data by optimizing a time-series database and implementing shortcuts. Here are some examples.Improved data ingestion performanceTime-series data is often generated at a high granularity. Going back to the previous finance example, there might be hundreds of data points generated every second or even more (depending on how many symbols you monitor). High volumes of data can be challenging to ingest efficiently. Time-series databases are equipped with internal optimizations like auto-partitioning and indexing, allowing you to scale up your ingestion rate.Simplified queryingWhen you query time-series data, you are often interested in how the data have been changing in the past five minutes, five days, five weeks, etc. You are also more likely to want to perform time-based calculations like time-based aggregations (time bucketing),moving averages,time-weighted averages,percentiles, and so on. A time-series database has specialized features to simplify and speed up the calculation of typical time-series queries.timesymbolprice2023-02-06 15:05:26PFE44.192023-02-06 15:05:25WMT141.272023-02-06 15:05:24KO59.672023-02-06 15:05:24TSLA194.082023-02-06 15:05:24SNAP10.88Store real-time and historical data in one placeThe value of time-series data comes from the fact that you can compare the most current state of your system with past situations. A time-series database has the tools and scale needed to store both historical (archives) and real-time data in one data store, making it easy for you to keep an eye on what’s happening right now while also having the ability to learn from the past. This gives you the power to simplify your data infrastructure and seamlessly analyze all your data in one place.Automated data managementTime-series databases can also automate your time-based data management tasks. For example, you might want to get rid of all data that is older than one year to save disk space and because you don’t need old data anymore. Or you still need to keep old data around, but maybe it’s not as handy as more recent data, you can set up your database to automatically compress old data to save on storage costs.There might be other valuable automations in a time-series database. For example, in TimescaleDB you can usecontinuous aggregatesto incrementally add data to a predefined materialized view—improving query performance and developer productivity.Top Time-Series DatabasesAccording to DB-Engines, here are some of the top time-series databases:InfluxDBKdbPrometheusGraphiteDolphinDBRRDtoolApache DruidTips for Choosing a Time-Series DatabaseOnce your applications start storing time-series data, you still have to pick a TSDB that best fits your data model, write/read pattern, and developer skill sets.When evaluating database options, consider these factors:Scalability. You must ensure that the database can scale vertically (adding more resources to a database node) and horizontally (adding more database nodes to your system) while remaining performant and reliable.Maintainability. Consider the time and effort it will take to maintain it long-term—backups, replicas, data retention, archiving, automation, and so on. Think about the maintenance jobs you want to do and see if the database has the tools and features to help you.Reliability. You might notice that many companies in this space are developing brand-new technologies from scratch.As time-series data quickly becomes the basis of business decisions and forecasts, you need to be sure that it will be available when you need it.Query language.There are also quite a few time-series databases on the market with their own custom languages (e.g., Flux by InfluxDB). This can be a turn-off for many developers because they’d need to learn a new language just to use the database. And even though some of them try to look like SQL,they are not real SQL.Time-Series Database Comparison: InfluxDB vs TimescaleDBTime-series databases may provide general features as well as time-series features (e.g., TimescaleDB) or focus on providing time-series features at the cost of supporting more general workloads (e.g., InfluxDB).Time-series data doesn’t live in a silo, so whatever the approach, you will need a way to relate general and time-series data.In the following section, you will see examples of time-series features you can expect from a time-series database. To show you both ends of the spectrum, you’ll see what the features look like in two totally different time-series databases: InfluxDB (with its own “Flux” query language) and TimescaleDB (which extends PostgreSQL and offers full-SQL support).‌‌📝 Editor's note:This article was written before InfluxDB 3.0.Read this more recent article about InfluxDB.Time-based aggregationTime-based aggregation is a must-have feature for time-series databases. Without it, you won’t be able to create hourly, daily, weekly, and monthly time aggregations (and many permutations in between) that are key foranalyzing time-series data.Let’s see what time-based aggregate queries look like in the case of InfluxDB and TimescaleDB:InfluxDBfrom(bucket: ""crypto"")
  |> range(start: 0)
  |> aggregateWindow(every: 1d, fn: count, createEmpty: false)
  |> sort(columns: [""_time"", ], desc:true)
  |> limit(n:10)Using InfluxDB’s query language, Flux, you must define a time range for the query. You can use 0 as the start parameter to work around this—in case you don’t want to define a specific time range. Then you can use theaggregateWindow()function to create arbitrary time “buckets”.TimescaleDBSELECT
  time_bucket('1 day', time) AS bucket,
  count(*) row_count
FROM crypto_ticks
GROUP BY bucket
ORDER BY bucket DESC
LIMIT 10;

bucket         	|row_count|
-------------------+---------+
2022-09-09 02:00:00|	52277|
2022-09-08 02:00:00|   128657|
2022-09-07 02:00:00|   134849|
2022-09-06 02:00:00|   132837|
2022-09-05 02:00:00|   126254|In TimescaleDB you can use thetime_bucket()function to create arbitrary time buckets. Besides, all the other available PostgreSQL functions like count(*) works the same way as in regular PostgreSQL.When working with time-series data, you will have to create a lot of time-based aggregations—so make sure that the database you choose provides a simple and intuitive interface to create time buckets.Time-series data modelingTime-series data, as mentioned, is always connected to time. For this reason, if you use a relational database to store time series, it makes sense to put an index on the time column. You can also create indexes that include the time and other columns you frequently filter.Indexes in relational databases are essential to improve performance. Some time-series databases provide this functionality by default, for example, when you create ahypertablein TimescaleDB, a time-series database. Let’s see an example data model for a typical financial application in PostgreSQL (the same would apply in TimescaleDB):CREATE TABLE stocks_real_time (
    time TIMESTAMPTZ NOT NULL,
    symbol TEXT NOT NULL,
    price DOUBLE PRECISION NULL
);This is a simplified version of our schema in ourgetting started guide. This example has one TIMESTAMPTZ column called time. If you use a relational database, it might make sense to put an index on this column to speed up queries where you filter by time (e.g., get me all the data from the past hour).The second column in this example is a symbol. This stores the company’s stock ticker symbol. You might consider creating an index that includes both time and symbol columns to improve the performance of queries where you filter by both time and symbol (e.g., get me all data for “TSLA” from the past hour).Finally, the last column in this example is price, which stores the price of the given stock symbol at the given time.Some databases only allow you to store timestamped data in your database. In contrast, others will enable you to store non-timestamped data right next to your timestamped data in the same database. This can be an important feature of the database if you want to simplify your data infrastructure and use only one database to store all of your data.Automatic downsamplingTime-series data is often ingested at a very high resolution (e.g., thousands of data points per second). To make it easier to analyze time series, users often downsample their data (e.g., they convert thousands of data points per second to only one). This technique doesn’t only save storage costs because you need to store lower-resolution data, but it alsomakes it easier to create visualizationsand recognize trends in the data.Downsampling is often done repeatedly and continuously, which means that if, for example, you insert multiple new rows every second, the database rolls up the incoming data into larger buckets automatically. Instead of aggregating the raw data yourself, the database takes care of it automatically and in real time.Let’s see how InfluxDB and TimescaleDB handle downsampling with anOHLCexample.InfluxDBclose=from(bucket: ""crypto"")
  |> range(start: -30d)
  |> group(columns:[""symbol""])
  |> filter(fn: (r) => r[""_measurement""] == ""ohlc"")
  |> window(every: 1h)
  |> reduce(fn: (r, accumulator) => ({
  
      indexLow:
        if (r._field==""low"") then 
          accumulator.indexLow+1 
        else
        accumulator.indexLow,
      indexOpen: 
      if (r._field==""open"") then 
       accumulator.indexOpen+1 
       else 
       accumulator.indexOpen,
        open: 
      if (r._field==""open"") then 
        if (accumulator.indexOpen==0) then 
          r._value 
        else 
          accumulator.open
      else
        accumulator.open  
    ,
    
    
      high:
       if (r._field==""high"") then  
          if(r._value>accumulator.high ) then
            r._value
          else
            accumulator.high 
      else 
        accumulator.high
   ,
    low: 
      if (r._field==""low"") then

          if(r._value<accumulator.low or accumulator.indexLow==0.0) then
            r._value
          else
           accumulator.low 
      else 
        accumulator.low,

             close: 
       if (r._field==""close"") then 
          r._value 
      else 
        accumulator.close,
             volume: 
        if (r._field==""volume"") then
          r._value+accumulator.volume 
          else
           accumulator.volume
             }),
    identity: {indexLow:0,indexOpen:0,open: 0.0,high: 0.0,low: 0.0,close: 0.0,volume: 0.0})
    |> drop(columns: [""indexOpen"",""indexLow""])
  |> group(columns:[""pair""])
    |> yield(name: ""candle"")InfluxDB’s Flux provides a convenient way to write simple queries, but if you want to create somewhat more complex queries, like creating OHLC aggregates from raw financial tick data, the final query can become quite long as you can see.TimescaleDBCREATE MATERIALIZED VIEW hourly_buckets
WITH (timescaledb.continuous)
AS
SELECT
  time_bucket('1 hour', time) AS bucket,
  symbol,
  first(price, time) AS open,
  max(price) AS high,
  min(price) AS low,
  last(price, time) AS close
FROM crypto_ticks
GROUP BY bucket, symbol;



SELECT * FROM hourly_buckets;
bucket         	|symbol  |open   |high   |low	|close  |
-------------------+--------+-------+-------+-------+-------+
2022-02-08 22:00:00|ADA/USD |  1.166|   1.17|  1.157|  1.168|
2022-02-08 22:00:00|ATOM/USD|  30.44|  30.63|   30.3|  30.51|
2022-02-08 22:00:00|AVAX/USD|  87.85|   88.0|  86.72|  87.06|
2022-02-08 22:00:00|BNB/USD |  413.5|  416.5|  410.3|  410.3|
2022-02-08 22:00:00|BTC/USD |44192.4|44354.0|43938.6|44185.2|‌If you are familiar with PostgreSQL syntax, you can see that the TimescaleDB method is very similar toa PostgreSQL materialized view. However, the mechanism under the hood is different to provide a better developer experience for time-series data by automatically storing pre-aggregated buckets over time, maintaining aggregations when raw data changes, and even returning real-time data.Querying recent dataYou might want to build visual dashboards to display time-series trends or even close to real-time data. For creatingtrend charts, you can use the previously mentioned downsampling method. But for real-time data, you probably want to see more granular and recent data, e.g., all data points from the past five minutes. Let’s see how you can make this simple request in InfluxDB and TimescaleDB.InfluxDB‌from(bucket: ""crypto"")
  |> range(start: -5m)
  |> filter(fn: (r) => r.symbol == ""BTC/USD"")
  |> sort(columns: [""_time"", ], desc:true)
  |> limit(n:5)
  |> keep(columns: [""_time"", ""_value""])In Flux, you can specify a time range that is relative to now with start: -5m, which will return all data for the “BTC/USD” symbol from the past five minutes.TimescaleDBSELECT
  time,
  price
FROM crypto_ticks
WHERE
  ""time"" > NOW() - INTERVAL '5 minutes' AND
  symbol = 'BTC/USD'
ORDER BY time DESC
LIMIT 5;

time           	|price  |
-------------------+-------+
2022-09-12 15:24:07|22346.7|
2022-09-12 15:24:03|22346.3|
2022-09-12 15:23:50|22346.7|
2022-09-12 15:23:45|22355.9|
2022-09-12 15:23:40|22358.1|In the TimescaleDB example, you can see a familiar SQL example (if you already know SQL) with a symbol filter and a relative time filter in the WHERE clause using the NOW() PostgreSQL function.Under the hood, the way this query gets executed is different from regular PostgreSQL, though: when you insert time-series data into the database, TimescaleDB auto-partitions your table based on the time column.Then, when you make a query containing a time filter, like in this example, TimescaleDB canexclude whole chunksfrom scanning which makes querying recent data lightning fast, even if you have billions of rows stored in the database.Long-range analytical queriesWhat if you are also interested in analyzing longer time frames, e.g. all data from the past year? Maybe you want to see what was the highest price of a certain stock or crypto symbol in the past year.InfluxDBfrom(bucket: ""crypto"")
  |> range(start: -1y)
  |> group(columns: [""code""])
  |> max()
  |> group()
  |> sort(columns: [""_value""], desc: true)This example shows that Flux executes your query in the same order as you describe it.TimescaleDBSELECT
  symbol,
  MAX(price) AS max_price
FROM crypto_ticks
WHERE
  ""time"" >= NOW() - INTERVAL '1 year'
GROUP BY symbol
ORDER BY max_price DESC;

symbol   |max_price |
---------+----------+
BTC/USD  |   48210.1|
WBTC/USD |  48169.56|
ETH/USD  |   3579.38|
BNB/USD  | 	460.0|
SOL/USD  |	143.55|Analytical queries like this, with a larger time window as the filter, are not typical time-series queries, but you might want to run these from time to time.TimescaleDB provides two features that significantly speed up these queries:native compression, which saves space and converts your data into a columnar form, andcontinuous aggregates, which automatically maintain materialized aggregate data that can be retained separately from raw readings.Together, these features can have a dramatic effect on the performance of your application.JOINing time-series data with other business dataSometimes we only talk about time-series data without mentioning all the other data that real-world projects have in their data infrastructure. But the reality is that time-series data is always connected to non-time-series (business) data.If you plan to analyze your time-series data and business data together, the database you choose needs to be able to JOIN them and work with them quickly and simply. In the following examples, you can see how to JOIN two tables in InfluxDB and TimescaleDB.InfluxDBcrypto_assets = from(bucket: ""crypto-assets"")
    |> range(start: -1mo)
    |> filter(fn: (r) => r._measurement == ""assets"" and r._field == ""symbol"")

crypto_ticks = from(bucket: ""crypto-ticks"")
    |> range(start: -1mo)
    |> filter(fn: (r) => r._measurement == ""ticks"" and r._field == ""price"")

join(
    tables: {assets:crypto_assets, ticks:crypto_ticks},
    on: [symbol, ],
)The big difference between InfluxDB and TimescaleDB in this regard is that InfluxDB can only store timestamped data, while TimescaleDB can store timestamped and non-timestamped data right next to each other. Thus, in InfluxDB you can only join time-series data with other time-series data but not relational data.TimescaleDBSELECT  crypto_assets.name,  bucket,  close,  high,  low,  open
FROM one_day_candle
INNER JOIN crypto_assets ON crypto_assets.symbol = one_day_candle.symbol
WHERE
  bucket > NOW() - INTERVAL '1 month' AND
  one_day_candle.symbol = 'BTC/USD'
ORDER BY bucket;

name   	|bucket         	|close  |high   |low	|open   |
-----------+-------------------+-------+-------+-------+-------+
Bitcoin USD|2022-08-13 02:00:00|24460.6|24889.5|24312.3|24402.2|
Bitcoin USD|2022-08-14 02:00:00|24312.4|25034.2|24160.4|24455.2|
Bitcoin USD|2022-08-15 02:00:00|24092.8|25210.9|23798.7|24316.2|
Bitcoin USD|2022-08-16 02:00:00|23867.7|24247.5|23692.0|24103.2|
Bitcoin USD|2022-08-17 02:00:00|23340.1|24430.1|23184.4|23857.3|In TimescaleDB, you can use PostgreSQL’s JOIN to connect any two tables in the same database, enabling you to store your non-time-series data next to your time-series data. Without this feature, you might have a harder time bringing your data together from multiple sources.Fun fact: One of the reasons TimescaleDB was created was that the founders struggled to find a database that could do easy JOINs for time-series data.Data retention and compressionWhile seeing how thousands of companies handle time-series data, we found that time-series data becomes less valuable over time. This means users often want to archive or even remove older data after a certain time to save on storage costs.InfluxDBIn InfluxDB you can change the data retention settings on a per-bucket basis on the UI.In older versions of InfluxDB, you could also add data retention policies this way:CREATE RETENTION POLICY ""one_year"" ON ""crypto_ticks"" DURATION 1y REPLICATION 1 DEFAULTTimescaleDB--Compression:
ALTER TABLE example SET (
  timescaledb.compress,
  timescaledb.compress_segmentby = symbol
);
SELECT add_compression_policy('crypto_ticks', INTERVAL '2 weeks');

--Data retention:
SELECT add_retention_policy('crypto_ticks', INTERVAL '1 year');With TimescaleDB, you can set up both a compression policy (to save on storage needs but keep the data available for querying) and a data retention policy (which gets rid of the data after the defined time period). Without essential tooling around data compression and data retention in the database, you’d need to implement and maintain these automations manually.Time-Series Database ResourcesIf you are interested in how other developers are using TimescaleDB for time series and analytics, check out our developer stories:How Octave Achieves a High Compression Ratio and Speedy Queries on Historical Data While Revolutionizing the Battery MarketHow Newtrax Is Using TimescaleDB and Hypertables to Save Lives in Mines While Optimizing ProfitabilityHow to Reduce Query Cost With a Wide Table Layout in TimescaleDBIf you want to keep reading about TimescaleDB, these articles will help you learn more:An introduction to hypertablesMore time-series data, less lines of code: meet hyper functionsUsing materialized views for time-series data analysisAllowing DML operations in highly-compressed data via Timescale compressionTimescaleDB performance benchmark vs Amazon RDS PostgreSQLFAQsWhat are the benefits of time-series databases?Time series databases are designed to efficiently store and query large volumes of time-series data, making them ideal for high-frequency data like IoT sensor data, server metrics, or financial data. With their optimized storage structures and compression algorithms, TSDBs offer faster write and query performance for time-based queries compared to traditional databases. They also often provide flexible schemas, allowing for easy addition or alteration of data fields. Furthermore, TSDBs can handle high data ingest rates and are built to scale horizontally, ensuring they can manage increased data loads effectively. Their specialized querying capabilities also make it simpler and more efficient to perform complex time-based analytics.How do time-series databases handle large volumes of data?Time-series databases handle large volumes of data by using optimized storage structures, compression algorithms, and indexing strategies. These elements allow for efficient storage and quick retrieval of time-stamped data. Additionally, many time-series databases are designed to scale horizontally, meaning they can distribute data across multiple servers to manage increased data loads effectively.What are the challenges in working with time-series data?Working with time-series data presents several challenges. These include handling the high volume and velocity of data, as time-series data is often generated continuously and in large quantities. Storing and querying this data efficiently can be difficult. Additionally, managing the real-time nature of the data, ensuring data quality, dealing with seasonality and time-dependent trends, anddeveloping accurate forecasting modelscan also be challenging.How do time-series databases work with real-time data?Time-series databases are designed to handle real-time data effectively. They can ingest high volumes of data rapidly and continuously, often providing near real-time insights. Their optimized storage and indexing strategies allow for fast data writes and quick retrieval of recent data.Some also offer stream processing or real-time analytics capabilities, enabling users to analyze and respond to trends in the data as it arrives.Get Started With TimescaleIf you're running your PostgreSQL database in your own hardware,you can simply add the TimescaleDB extension. If you prefer to try Timescale in AWS,create a free account on our platform. It only takes a couple seconds, no credit card required.📝 Are you migrating off InfluxDB? We can help.Check outOutflux, our migration tool, andreach out to us for expert migration advice.Our Support team has helped many others migrate from InfluxDB to Timescale successfully!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/what-is-a-time-series-database/
2022-02-23T14:23:08.000Z,Increase Your Storage Savings With TimescaleDB 2.6: Introducing Compression for Continuous Aggregates,"Yesterday, we announced thatwe raised $110 million in our Series C. 🐯🦄🚀 Today, we keep celebrating the big news Timescale-style: with an #AlwaysBeLaunching spirit! We're excited to announce the release of TimescaleDB 2.6, a version that comes with new features highly requested by our community - most notablycompression for continuous aggregates. 🔥 TimescaleDB 2.6 also includes theexperimentalsupport fortimezones in continuous aggregates.We love building in public. We are firm believers in the value of user feedback, as there’s no better way to improve your product than hearing from those who use it daily. We read all the thoughts and comments you share in our Community Slack, and we pay close attention toyour feature requests in GitHub. When features get upvoted, it helps us prioritize what to work on next.With today’s release, we are proud to bring you a top requested feature:the support for compression in continuous aggregates. Originally, we envisioned that compression would mostly be necessary over raw data, as continuous aggregates by themselves help downsample datasets considerably. But TimescaleDB users operate at such a scale thatthey requested compression also for their continuous aggregatesto save even more disk space.Theissuethat originated the support for compression in continuous aggregatesContinuous aggregatesspeed up aggregate queries over large volumes. You can think of them as a more powerful version of PostgreSQL materialized views, as they allow you to materialize your data while your view getsautomatically and incrementally refreshed in the background. But continuous aggregates are also very useful for something else: downsampling. Indeed, another property of continuous aggregates is that you can keep them around even when the data from the underlying hypertable has been dropped. This allows you to reduce the granularity of your data once it reaches a certain age, liberating space while still enabling long-term analytics.The ability to compress continuous aggregates takes this one step further. Starting with TimescaleDB 2.6, you can apply TimescaleDB’snative columnar compressionto your continuous aggregates, freeing even more disk space. And by combiningcompression policies(which automatically compress data after a certain period of time) withdata retention policies, you can automatically set up a downsampling strategy for your older data.Figure describing the downsampling process through continuous aggregates and compression. Notice the relationship between the refresh policy, compression policy, and data retention policy.TimescaleDB 2.6 also comes with another highly requested feature by the community:you're now able to create continuous aggregates with monthly buckets and/or timezonesusingtime_bucket_ng. (Note that this is anexperimentalfeature.)time_bucket_ngis the “new generation” of ourtime_buckethyperfunction, used for bucketing and analyzing data for arbitrary time intervals in SQL. You can think oftime_bucketas a more powerful version of thePostgreSQLdate_truncfunction, allowing for arbitrary time intervals rather than the standard day, minute, hour provided bydate_trunc. Buttime_bucketdoesn’t yet support buckets by months, years, or timezones;time_bucket_ngexpands the capabilities oftime_bucketby including these features in our experimental schema. (We first introduced this featurethanks to an issue from the community.)We’re eager to hear how this works for you. Our main goal with experimental features is to get as much input as possible - please, if you see something that needs improvements,tell us in GitHub!Theissuethat originated the support for timezones in continuous aggregates.​​TimescaleDB 2.6 is available today. If you are already a TimescaleDB user,check out our docs for instructions on how to upgrade. If you are using Timescale Cloud, upgrades are automatic, and no further action is required from your side (you can also start a free 30-day trial, no credit card required).If you are new to Timescale and you want to learn more about continuous aggregates, compression, andtime_bucket_ng, keep reading!Once you’re using TimescaleDB, join our community. You can ask us questions in ourCommunity Slackor in our brand newCommunity Forum, a better home for long-form discussions. We’ll be more than happy to solve any doubts you may have about compression and continuous aggregates, TimescaleDB 2.6, or any other topic. And if you share our mission of helping developers worldwide, we arehiring broadly across many roles!Before moving on, a huge thank you to the team of engineers (and the entire team of reviewers and testers) that made this release possible. This includes the community members who helped us prioritize, develop, and test these features. As we’ve said before, none of this would be possible without your requests, your upvotes, and your feedback.Please, keep ‘em coming!The power of continuous aggregatesContinuous aggregates in TimescaleDBare a more powerful version ofPostgreSQL materialized views. For example, continuous aggregates are incrementally updated with a built-in refresh policy, so they stay up-to-date as new data is added. When querying a continuous aggregate,the query engine will combine the data that is already pre-computed in the materialized view with the newest raw data in the underlying hypertable. In other words, you will always see up-to-date results.TimescaleDB uses internal invalidation records to determine which data has been changed or added in the underlying hypertable since the last refresh; when you refresh your continuous aggregate,only the new or updated data is computed.This means that TimescaleDB doesn’t need to look at your whole table every time you do a refresh, saving tons of computation resources, speeding up the incremental maintenance of continuous aggregate, and allowing you to get faster results in your aggregate queries.This also implies that you can also use continuous aggregates for downsampling. Differently than with materialized views, you are able to retain the continuous aggregate even after the original raw data has been dropped. To reduce the granularity of your time-series data, you can simply define a continuous aggregate and delete your original data once it gets old - we will be demonstrating this process later in this blog post.Compression in TimescaleDBTogether with continuous aggregates,compressionis another oft-used feature of TimescaleDB. By defining acompression policy, TimescaleDB allows you to compress all data stored in a specific hypertable that is older than a specific period of time - let’s say, 7 days old. In this example, you would keep your last week’s worth of data uncompressed, which would allow you to write data into your database at very high rates, giving you optimal query performance for your shallow-and-wide queries as well. Once your data gets old enough (after 7 days), the compression policy would kick in, automatically compressing your data.TimescaleDB is able to give you high compression rates by deployingbest-in-class compression algorithmsalong with anovel hybrid row/columnar storage design. Once a chunk inside your hypertable becomes old enough, TimescaleDB compresses it by storing the columns into an array. In other words: your more recent chunks (in the previous example, all chunks newer than 7 days) will be stored in TimescaleDB as relational, row-based partitions. Once compressed, your chunks will be a columnar store.An example chunk before compression vs after compression. After compression, the data previously stored in multiple rows is now stored as a single row, with the columns being stored as an array.✨ If you want to dive deeper into how compression works in TimescaleDB, check out the following videos:Compression 101 (part 1): learn all the fundamentals of compression and compression policies in TimescaleDB.Compression deep dive (part 2): expand your compression knowledge by getting familiar with more advanced functionality, like how to adequately use and configureSEGMENT BYcolumns.How to use continuous aggregates with compression for downsamplingAs we introduced earlier, continuous aggregates can help you reduce the granularity of your dataset, with the ultimate goal of saving some disk space.We explained previously how, in TimescaleDB, you are able to retain the continuous aggregate even after the original raw data has been dropped. So in order to considerably reduce your data size (and thus your storage costs) automatically and without losing the ability to do long term analytics on your older data, you can:Create a continuous aggregatecapturing the information that you most likely will like to see in your historical analytic queries.Define a refresh policy for your continuous aggregateso your continuous aggregate can stay up to date, periodically materializing your newest data.Enable compressionin the continuous aggregate.Add a compression policyto compress your chunks automatically once they’re older than a specific period of time.Create a data retention policyto automatically drop the raw data in the original hypertable once it gets older than a specific period of time.In this section, we’ll walk you through an example, using financial data fromAlpha Vantage. (If you want to also load this dataset into your TimescaleDB instance, we have astep-by-step guide published in our docs.)We will be using the following schema:CREATE TABLE public.stocks_intraday (
	""time"" timestamptz NOT NULL,
	symbol text NULL,
	price_open float8 NULL,
	price_high float8 NULL,
	price_low float8 NULL,
	price_close float8 NULL,
	trading_volume int4 null
	);
    
SELECT create_hypertable ('public.stocks_intraday', 'time');Create a continuous aggregate and set up continuous aggregate policyTo start, let’screate a continuous aggregateusing TimescaleDB’stime_bucket()function. We will record averages for the price at open, high, low, and close for each company symbol over a given day.We also will set up acontinuous aggregate policy, which will update and refresh data from the last four days. This policy will run once every hour:CREATE MATERIALIZED VIEW stock_intraday_daily
WITH (timescaledb.continuous, timescaledb.materialized_only = true) AS
SELECT 
time_bucket( interval '1 day', ""time"") AS bucket,
AVG(price_high) AS high,
AVG(price_open) AS open,
AVG(price_close) AS close,
AVG(price_low) AS low,
symbol 
FROM stocks_intraday si 
GROUP BY bucket, symbol;Enable and set up compression on your continuous aggregateNow that we have defined the continuous aggregate together with a refresh policy for it, we canenable compressionon this continuous aggregate, also setting up ourcompression policy. This compression policy will automatically compress all chunks older than 7 days:ALTER MATERIALIZED VIEW stock_intraday_daily SET (timescaledb.compress = true);

-- Set up compression policy
SELECT add_compression_policy('stock_intraday_daily', INTERVAL '7 days');Lastly, it is important to notice thatupdating data within a compressed chunk is not supported yet in TimescaleDB. This is relevant for correctly configuring compression policies in continuous aggregates: since refresh policies require chunks to be updated, we have to make sure that our recent chunks remain uncompressed. I.e., make sure you define your time intervals so that your continuous aggregate gets refreshed at a later date than when your compression policy is set.How compression affects storageSo we officially have a compressed continuous aggregate, but you may be wondering: how much of a difference does compression make on this continuous aggregate?In order to find out, let’s check out the stats on our compressed continuous aggregate. To do that, we first need to find the internal name for the materialized hypertable. We can do it by looking at our compressed tables:SELECT * FROM timescaledb_information.compression_settings;Then, we can use that to run thehypertable_compression_stats()command.(Note: The name of your materialized hypertable will most likely be different than the one shown below.)SELECT * FROM hypertable_compression_stats('_timescaledb_internal._materialized_hypertable_3');

Results:
—---------------------------------|--------
total_chunks                   | 7
number_compressed_chunks       | 6
before_compression_table_bytes | 7471104
before_compression_index_bytes | 2031616
before_compression_toast_bytes | 49152
before_compression_total_bytes | 9551872
after_compression_table_bytes  | 401408
after_compression_index_bytes  | 98304
after_compression_toast_bytes  | 3178496
after_compression_total_bytes  | 3678208
Node_nameFor this continuous aggregate, we got a compression rate of over 61%:Size of the continuous aggregate before and after compression (9.6 MB vs 3.7 MB)A 61% compression rate would imply a very nice boost in your storage savings. However, if you’re used to compression in TimescaleDB, you may be wondering: whyonly61%? Why cannot I getcompression rates over 90%as I commonly see with my hypertables?The reason behind this is that continuous aggregates store partial representation of the aggregates inbytea format, which is compressed using dictionary-based compression algorithms. So continuous aggregates cannot take advantage of other compression algorithms - at least, not yet. We’re experimenting with some concepts that may significantly increase the compression rates for continuous aggregates in the future. Stay tuned!Set up a data retention policy on your raw dataThrough the continuous aggregate we created earlier, we could effectively downsample our data. Our original dataset had one datapoint per minute; this is perfect for real-time monitoring, but a bit too heavy for the purpose of long-term analysis. In our continuous aggregate, we’re aggregating the data into 1 h buckets, which is a more manageable granularity if you’re planning to store this data long-term. So let’s save up some additional disk space by dropping the data in the underlying hypertable while keeping the continuous aggregate.To do so, we will define aretention policythat automatically deletes our raw data once it reaches a certain age. This age is completely up to you and your use case (the retention policy below will delete the data older than a month, for example). As a reminder, this policy refers only to the data in your original hypertable - you will still keep the data materialized in the continuous aggregate.SELECT add_retention_policy('stocks_intraday', INTERVAL '1 month');You asked, and we delivered: introducing timezones for continuous aggregatesAs we mentioned at the beginning of this post, TimescaleDB 2.6 introduces not only compression for continuous aggregates but also the possibility of using timezones in continuous aggregates,another highly requested feature by our community. Starting with TimescaleDB 2.6, you're able to create continuous aggregates with monthly buckets and/or timezones usingtime_bucket_ng.This is an experimental feature: please,test it and send us your feedback!The more feedback we get, the faster we will make this feature ready for production. 🔥time_bucket_ngis the “new generation” of ourtime_buckethyperfunction,  used for bucketing and analyzing data for arbitrary time intervals in SQL. You can think oftime_bucketas a more powerful version of thePostgreSQLdate_truncfunction, allowing for arbitrary time intervals rather than the standard day, minute, hour provided bydate_trunc.Buttime_bucketdoesn’t support time buckets by months, years, or timezones.time_bucket_ngexpandstime_bucketby including these features, giving users maximum flexibility in their queries. (We also first introducedtime_bucket_ngthanks to an issue from the community!)time_bucket_ngis still an experimental feature, being that it’s still being developed underour experimental schema. In Timescale, we like to bring constant value to our users by moving fast, but we also value stability - as we like to say, wemove fast without breaking things. Through our experimental schema, we’re allowed to release experimental features that, even if they’re still not ready for production, are ready to be widely tested. As we keep mentioning through this post, we love to get as much feedback from the community as possible: releasing features under experimental helps us build robust features, as by the time we “graduate” these features out of the experimental schema, we are sure that everything is stable.Starting with TimescaleDB 2.6, you can now use time buckets of months and years plus specify timezones in your continuous aggregates. For example, the continuous aggregate below tracks the temperature in Honolulu over monthly intervals.(Yes, we are dreaming of warmth 🏝 in this team!)CREATE TABLE conditions(
  day timestamptz NOT NULL,
  city text NOT NULL,
  temperature INT NOT NULL);

SELECT create_hypertable(
  'conditions', 'day',
  chunk_time_interval => INTERVAL '1 day'
);

INSERT INTO conditions (day, city, temperature) VALUES
  ('2021-06-14 00:00:00 HST', 'Honolulu', 26),
  ('2021-06-15 00:00:00 HST', 'Honolulu', 22),
  ('2021-06-16 00:00:00 HST', 'Honolulu', 24),
  ('2021-06-17 00:00:00 HST', 'Honolulu', 24),
  ('2021-06-18 00:00:00 HST', 'Honolulu', 27),
  ('2021-06-19 00:00:00 HST', 'Honolulu', 28),
  ('2021-06-20 00:00:00 HST', 'Honolulu', 30),
  ('2021-06-21 00:00:00 HST', 'Honolulu', 31),
  ('2021-06-22 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-23 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-24 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-25 00:00:00 HST', 'Honolulu', 32),
  ('2021-06-26 00:00:00 HST', 'Honolulu', 32),
  ('2021-06-27 00:00:00 HST', 'Honolulu', 31);

CREATE MATERIALIZED VIEW conditions_summary
WITH (timescaledb.continuous) AS
SELECT city,
   timescaledb_experimental.time_bucket_ng('1 month', day, 'Pacific/Honolulu') AS bucket,
   MIN(temperature),
   MAX(temperature)
FROM conditions
GROUP BY city, bucket;

-- to_char() is used because timestamptz is displayed in the sesison timezone by default
-- alternatively you can use SET TIME ZONE 'Pacific/Honolulu';
SELECT city, to_char(bucket at time zone 'HST', 'YYYY-MM-DD HH24:MI:SS') as month, min, max
FROM conditions_summary
ORDER by month, city;

   city   |        month        | min | max
----------+---------------------+-----+-----
 Honolulu | 2021-06-01 00:00:00 |  22 |  34
(1 row)We’re eager to know how this feature is working for you so we can improve it. Please, reach out to us throughGitHub, ourCommunity Slack, or theTimescale Community Forum.And thank you again for your invaluable support!Get startedTimescaleDB 2.6 is already available for Timescale Cloud and self-managed TimescaleDB:If you are a Timescale Cloud user, you will be automatically upgraded to TimescaleDB 2.6during your next maintenance window. No action is required from your side. You can also create a free Timescale Cloud account to get a free 30-day trial, with no credit card required.If you are using TimescaleDB in your own instances,check out our docs for instructions on how to upgrade.If you are using Managed Service for TimescaleDB, TimescaleDB 2.6 will be available for you in the upcoming weeks.Once you’re using TimescaleDB, connect with us! You can find us in ourCommunity Slackand theTimescale Community Forum. We’ll be more than happy to answer any question on continuous aggregates, compression, TimescaleDB, PostgreSQL, or anything in between.And if you want to help us build and improve features like compression, continuous aggregates, andtime_bucket,we are hiring broadly across many roles! Join our global, fully remote team. 🌎Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/increase-your-storage-savings-with-timescaledb-2-6-introducing-compression-for-continuous-aggregates/
2022-09-15T13:52:39.000Z,How We’re Raising the Bar on Hosted Database Support,"If choosing a database in a crowded market is challenging, choosing where you will run said database can prove equally tricky. The options are endless, but it’s pretty common these days to run your database on a hosted service, whether that’s something like Amazon RDS or a database company’s own cloud, likethe one we have at Timescale. Still, one of the most critical aspects to consider when selecting a hosted provider is cloud support. What will your cloud provider do to help ensure your hosted database is successful?The Value of Deep, Consultative SupportFrom the user’s perspective, one should expect a minimum set of services from the hosting provider, from backups to speedy and steady help when the database is down or other platform-related issues arise, etc. But what if your queries are slow? Or what if a feature does not work as expected? Or would you like guidance on how to architect for future growth? You may not have to consider the physical infrastructure when running a cloud database, but these are still real and valid concerns.Here at Timescale, we want you to be successful with your time-series and relational data. One of the ways we do that is through ourSlack Community, where many of us here at Timescale work together with many folks in our community on questions and issues. Many folks from our community work with each other as well. It’s a fantastic, vibrant community and is open to everyone. We also have a very activeTimescale Community Forum. But that’s not all we do—we are raising the bar on hosted database support.For our users on Timescale, we take it one step further. We have a team of support engineers located around the world to help with migrations, data modeling, query or ingest performance, compression settings, and more.We offer deep, consultative support for every Timescale user at no additional charge.This is an investment we’re making in our users, just as our users are making an investment in us by choosing us to host their time-series and relational data. As a comparison, with Amazon RDS,deeply consultative support in addition to general guidance and best practices starts at over $5,000 per month, and lower tiers have only a community forum or only offer general advice.Here’s what you can expect from working with the Timescale Support Team.Every Case Is UniqueAdopting a new technology (or refining the operations of the one you’re already using) is hard. While TimescaleDB is an extension on top of PostgreSQL, making it much more immediately recognizable and reducing the amount of new information to learn, we still have our own lingo and concepts.Hypertablesandcontinuous aggregatesandcompression. Oh, my!The thing about doing support for a product like TimescaleDB is that there often aren’t easy answers. This is actually a good thing! Our users are talented professionals. Because of that, no easy answers means that the things that do have easy answers—basic configurations, API conventions, etc.—are either well documented or intuitive enough that folks have already figured them out.What gets left over are the less clear-cut questions and scenarios: Why did this query perform poorly? What should I use for segmenting in compression based on X, Y, and Z factors? Should I use one hypertable or many hypertables?✨Editor's Note:One of the most common questions is optimizing “chunk” size. (“Chunk” is Timescale lingo for data partitions within a table.)We answered this question in this blog post.These questions have many variables, requiring some back and forth. That is the key to our support here at Timescale: collaboration. We don’t tend just to give rote answers.Because when it comes to data, there is no one-size-fits-all strategy. We work with you to understand your problem and requirements and devise a solution that works for you in your specific circumstances.Our approach is inquisitive and exploratory because we want to answer not only the question you have now but the ones you don’t know you have yet.Work With Your Customers, Improve Your ProductWe believe strongly in working through questions and answers together. We’re all subject to the unknown unknowns of theJohari window, so by working together, we collectively increase the things we know and decrease the things we don’t. We understand that when you’re looking for help, the last thing you probably want to get back is a bunch of questions. We ask them to gain a more holistic understanding of the issue at hand to provide more than a bandaid but a real, lasting solution.Through our interactions with individual customers, we improve our product for everyone.We view every support case as an opportunity to learn, both as individuals and as a company. As a company, your cases tell us where we can improve. If something isn’t working, that’s something we should look into making better. As individuals, we learn new and innovative ways to look at, analyze, collect, and use data. We learn about operational models and DevOps practices. We learn about new and exciting technologies that work alongside our own that help you answer the questions you need to ask of your data.In support, at our core, we are eternally curious. Working with you gives us the opportunity to explore new and different things every day. Along the way, we hope we can both learn something as we work through your questions together. At the end of the day, this interchange of ideas makes us all better.Work With Us!We’d love to work through some of your questions together. If you are a current user ofTimescale, you know where we’re at—shoot us a message at[email protected].If you’re not yet a Timescale user,you can use it for free for 30 days, no credit card required—and you have full access to our support team during your trial. We look forward to working with you!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-were-raising-the-bar-on-hosted-database-support/
2021-12-01T14:59:56.000Z,PostgreSQL vs Python for Data Cleaning: A Guide,"IntroductionDuring analysis, you rarely—if ever—get to go directly from evaluating data to transforming and analyzing it. Sometimes to properly evaluate your data, you may need to do some pre-cleaning before you get to the main data cleaning, and that’s a lot of cleaning! In order to accomplish all this work, you may use Excel, R, or Python, but are these the best tools for data cleaning tasks?In this blog post, I explore some classicdata cleaningscenarios and show how you can perform themdirectly within your databaseusingTimescaleDBandPostgreSQL, replacing the tasks that you may have done in Excel, R, or Python. TimescaleDB and PostgreSQL cannot replace these tools entirely, but they can help your data munging/cleaning tasks be more efficient and, in turn, let Excel, R, and Python shine where they do best: in visualizations, modeling, and machine learning.Cleaning is a very important part of the analysis process and generally can be the most grueling from my experience! By cleaning data directly within my database, I am able to perform a lot of my cleaning tasks one time rather than repetitively within a script, saving me considerable time in the long run.A recap of the data analysis processI began this series of posts ondata analysisby presenting the following summary of the analysis process:Data Analysis LifecycleThe first three steps of the analysis lifecycle (evaluate, clean, transform) comprise the “data munging” stages of analysis. Historically, I have done my data munging and modeling all within Python or R, these being excellent options for analysis. However, once I was introduced to PostgreSQL and TimescaleDB, I found how efficient and fast it was to do my data munging directly within my database. In my previous post, I focused on showingdata evaluationtechniques and how you can replace tasks previously done in Python with PostgreSQL and TimescaleDB code. I now want to move on to the second step,data cleaning. Cleaning may not be the most glamorous step in the analysis process, but it is absolutely crucial to creating accurate and meaningful models.As I mentionedin my last post, my first job out of college was at an energy and sustainability solutions company that focused on monitoring all different kinds of utility usage - such as electricity, water, sewage, you name it - to figure out how our clients’ buildings could be more efficient. My role at this company was to perform data analysis and business intelligence tasks.Throughout my time in this job, I got the chance to use many popular data analysis tools including Excel, R, and Python. But once I tried using a database to perform my data munging tasks - specifically PostgreSQL and TimescaleDB - I realized how efficient and straightforward analysis, and particularly cleaning tasks, could be when done directly in a database.Before using a database for data cleaning tasks, I would often find either columns or values that needed to be edited. I would pull the raw data from a CSV file or database, then make any adjustments to this data within my Python script. This meant that every time I ran my Python script, I would have to wait for my machine to spend computational time setting up and cleaning my data. This means that I lost time with every run of the script. Additionally, if I wanted to share cleaned data with colleagues, I would have to run the script or pass it along to them to run. This extra computational time could add up depending on the project.Instead, with PostgreSQL, I can write a query to do this cleaning once and then store the results in a table. I wouldn’t need to spend time cleaning and transforming data again and again with a Python script, I could just set up the cleaning process in my database and call it a day! Once I started to make cleaning changes directly within my database, I was able to skip performing cleaning tasks within Python and simply focus on jumping straight into modeling my data.To keep this post as succinct as possible, I chose to only show side-by-side code comparisons for Python and PostgreSQL. If you have any questions about other tools or languages, please feel free to join ourSlack channel, where you can ask the Timescale community, or me, specific questions about Timescale or PostgreSQL functionality 😊. I’d love to hear from you!Additionally, as we explore TimescaleDB and PostgreSQL functionality together, you may be eager to try things out right away! Which is awesome! The easiest way to get started is by signing up fora free 30-day trial of Timescale (if you prefer self-hosting, you can alwaysinstall and manage TimescaleDB on your own PostgreSQL instances). Learn more byfollowing one of our many tutorials.Now, before we dip into things and get our data, as Outkast best put it, “So fresh, So clean”, I want to quickly cover the data set I will be using. In addition, I also want to note that all the code I show will assume you have some basic knowledge of SQL. If you are not familiar with SQL, don’t worry! In my last post, I included a section on SQL basics which you can findhere.About the sample datasetIn my experience within the data science realm, I have done the majority of my data cleaning after evaluation. However, sometimes it can be beneficial to clean data, evaluate, and then clean again. The process you choose is dependent on the initial state of your data and how easy it is to evaluate. For the data set I will use today, I would likely do some initial cleaning before evaluation and then clean again after, and I will show you why.I got the followingIoT data set from Kaggle, where a very generous individual shared their energy consumption readings from their apartment in San Jose CA, this data incrementing every 15 minutes. While this is awesome data, it is structured a little differently than I would like. The raw data set follows this schema:and appears like this…typedatestart_timeend_timeusageunitscostnotesElectric usage2016-10-2200:00:0000:14:000.01kWh$0.00Electric usage2016-10-2200:15:0000:29:000.01kWh$0.00Electric usage2016-10-2200:30:0000:44:000.01kWh$0.00Electric usage2016-10-2200:45:0000:59:000.01kWh$0.00Electric usage2016-10-2201:00:0001:14:000.01kWh$0.00Electric usage2016-10-2201:15:0001:29:000.01kWh$0.00Electric usage2016-10-2201:30:0001:44:000.01kWh$0.00Electric usage2016-10-2201:45:0001:59:000.01kWh$0.00In order to do any type of analysis on this data set, I want to clean it up. A few things that quickly come to mind include:The cost is seen as a text data type which will cause some issues.The time columns are split apart which could cause some problems if I want to create plots over time or perform any type of modeling based on time.I may also want to filter the data based on various parameters that have to do with time, such as day of the week or holiday identification (both potentially play into how energy is used within the household).In order to fix all of these things and get more valuable data evaluation and analysis, I will have to clean the incoming data! So without further ado, let’s roll up our sleeves and dig in!Cleaning the dataI will show most of the techniques I have used in the past while working in data science. While these examples are not exhaustive, I hope they will cover many of the cleaning steps you perform during your own analysis, helping to make your cleaning tasks more efficient by using PostgreSQL and TimescaleDB.Please feel free to explore these various techniques and skip around if you need! There is a lot here, and I designed it to be a helpful glossary of tools that you could use as you need.The techniques that I will cover include:Correcting structural issuesCreating or generating relevant dataAdding data to a hypertableRenaming columns or tablesFill in missing valuesNote on cleaning approach:There are many ways that I could approach the cleaning process in PostgreSQL. I could create a table thenALTERit as I clean, I could create multiple tables as I add or change data, or I could work withVIEWs. Depending on the size of my data, any of these approachescouldmake sense, however, they will have different computational consequences.You may have noticed above that my raw data table was calledenergy_usage_staging. This is because I decided that given the state of my raw data, it would be best for me to place the raw data in astaging table, clean it usingVIEWs, then insert it into a more usable table as part of my cleaning process. This move from raw table to the usable table could happen even before the evaluation step of analysis. As I discussed above, sometimes data cleaning has to occur after AND before evaluating your data. Regardless, this data needs to be cleaned and I wanted to use the most efficient method possible. In this case, that meant using a staging table and leveraging the efficiency and power of PostgreSQLVIEWs, something I will talk about later.Generally, if you are dealing with a lot of data, altering an existing table in PostgreSQL can be costly. For this post, I will show you how to build up clean data usingVIEWs along with additional tables. This method of cleaning is more efficient and sets you up for the next blog post about data transformation which includes the use of scripts in PostgreSQL.Correcting structural issuesRight off the bat, I know that I need to do some data refactoring on my raw table due to data types. Notice that we havedateand time columns separated andcostsis recorded as a text data type. I need to convert my separated date time columns to a timestamp and thecostcolumn to float4. But before I show that, I want to talk about why conversion to timestamp is beneficial.TimescaleDB hypertables and why timestamp is importantFor those of you not familiar with the structure ofTimescaleDB hypertables, they are at the basis of how we efficiently query and manipulate time-series data. Timescale hypertables are partitioned based on time, and more specifically by the time column you specify upon creation of the table.The data is partitioned by timestamp into ""chunks"" so that every row in the table belongs to somechunkbased on a time range. We then use these time chunks to help query the rows so that you can get more efficient querying and data manipulation based on time. This image represents the difference between a normal table and our special hypertables.Changing date-time structureBecause I want to utilize TimescaleDB functionality to the fullest, such as continuous aggregates and faster time based queries, I want to restructure theenergy_usage_stagingtable'sdateand time columns. I could use thedatecolumn for my hypertable partitioning, however, I would have limited control over manipulating my data based on time. It is more flexible and space efficient to have a single column with a timestamp than it is to have separate columns with date and time. I can always extract the date or time from the timestamp if I want to later!Looking back at the table structure, I should be able to get a usable timestamp value from thedateandstart_timecolumns as theend_timereally doesn’t give me that much useful information. Thus, I want to essentially combine these two columns to form a new timestamp column, let’s see how I can do that using SQL. Spoiler alert, it is as simple as an algebraic statement. How cool is that?!PostgreSQL code:In PostgreSQL I can create the column without inserting it into the database just yet. Since I want to create a NEW table from this staging one, I don’t want to add more columns or tables just yet.Let’s first compare the original columns with our new generated column. For this query I simplyaddthe two columns together. TheASkeyword just allows me to rename the column to whatever I would like, in this case beingtime.--add the date column to the start_time column
SELECT date, start_time, (date + start_time) AS time 
FROM energy_usage_staging eus;Results:datestart_timetime2016-10-2200:00:002016-10-22 00:00:00.0002016-10-2200:15:002016-10-22 00:15:00.0002016-10-2200:30:002016-10-22 00:30:00.0002016-10-2200:45:002016-10-22 00:45:00.0002016-10-2201:00:002016-10-22 01:00:00.0002016-10-2201:15:002016-10-22 01:15:00.000Python code:In Python, the easiest way to do this is to add a new column to the dataframe. Notice that in Python I would have to concatenate the two columns along with a defined space, then convert that column to datetime.energy_stage_df['time'] = pd.to_datetime(energy_stage_df['date'] + ' ' + energy_stage_df['start_time'])
print(energy_stage_df[['date', 'start_time', 'time']])Changing column data typesNext, I want to change the data type of my cost column from text to float. Again, this is straightforward in PostgreSQL with theTO_NUMBER()function.The format of the function is as follows:TO_NUMBER(‘text’, ‘format’). The ‘format’ input is a PostgreSQL specific string that you can build depending on what type of text you want to convert. In our case we have a$symbol followed by a numeric set up0.00. For the format string I decided to use ‘L99D99’. The L lets PostgreSQL know there is a money symbol at the beginning of the text, the 9s let the system know I have numeric values, and then the D stands for a decimal point.I decided to cap the conversion on values that would be less than or equal to ‘$99.99’ because the cost column has no values greater than 0.65. If you were planning to convert a column with larger numeric values, you would want to account for that by adding in a G for commas. For example, say you have a cost column with text values like this ‘$1,672,278.23’ then you would want to format the string like this ‘L9G999G999D99’PostgreSQL code:--create a new column called cost_new with the to_number() function
SELECT cost, TO_NUMBER(""cost"", 'L9G999D99') AS cost_new
FROM energy_usage_staging eus  
ORDER BY cost_new DESCResults:costcost_new$0.650.65$0.650.65$0.650.65$0.570.57$0.460.46$0.460.46$0.460.46$0.460.46Python code:For Python, I used a lambda function that systematically replaces all the ‘$’ signs with empty strings. This can be fairly inefficient.energy_stage_df['cost_new'] = pd.to_numeric(energy_stage_df.cost.apply(lambda x: x.replace('$','')))
print(energy_stage_df[['cost', 'cost_new']])Creating aVIEWNow that I know how to convert my columns, I can combine the two queries and create aVIEWof my new restructured table. AVIEWis a PostgreSQL object which allows you to define a query and call it by it’sVIEWs name, as if it were a table within your database. I can use the following query to generate the data I want and then create aVIEWthat I can query it as if it were a table.PostgreSQL code:-- query the right data that I want
SELECT type, 
(date + start_time) AS time, 
""usage"", 
units, 
TO_NUMBER(""cost"", 'L9G999D99') AS cost, 
notes 
FROM energy_usage_stagingResults:typetimeusageunitscostnotesElectric usage2016-10-22 00:00:00.0000.01kWh0.00Electric usage2016-10-22 00:15:00.0000.01kWh0.00Electric usage2016-10-22 00:30:00.0000.01kWh0.00Electric usage2016-10-22 00:45:00.0000.01kWh0.00Electric usage2016-10-22 01:00:00.0000.01kWh0.00Electric usage2016-10-22 01:15:00.0000.01kWh0.00Electric usage2016-10-22 01:30:00.0000.01kWh0.00Electric usage2016-10-22 01:45:00.0000.01kWh0.00Electric usage2016-10-22 02:00:00.0000.02kWh0.00Electric usage2016-10-22 02:15:00.0000.02kWh0.00I decided to call myVIEWenergy_view. Now, when I want to do further cleaning, I can just specify its name in theFROMstatement.--create view from the query above
CREATE VIEW energy_view AS
SELECT type, 
(date + start_time) AS time, 
""usage"", 
units, 
TO_NUMBER(""cost"", 'L9G999D99') AS cost, 
notes 
FROM energy_usage_stagingPython code:energy_df = energy_stage_df[['type','time','usage','units','cost_new','notes']]
energy_df.rename(columns={'cost_new':'cost'}, inplace = True)
print(energy_df.head(20))It is important to note that with PostgreSQLVIEWs, the data inside of them have to be recalculated every time you query it. This is why we want to insert ourVIEWdata into a hypertable once we have the data set up just right. You can think ofVIEWs as a shorthand version of theCTEsWITHASstatement I discussed in my last post.We are now one step closer to cleaner data!Creating or generating relevant dataWith some quick investigation, we can see that the notes column is blank for this data set. To check this I just need to include aWHEREclause and specify wherenotesare not equal to an empty string.PostgreSQL code:SELECT * 
FROM energy_view ew
-- where notes are not equal to an empty string
WHERE notes!='';Results come out emptyPython code:print(energy_df[energy_df['notes'].notnull()])Since the notes are blank, I would like to replace the column with various sets of additional information that I could use later on during modeling. One thing I would like to add in particular, is a column that specifies the day of the week. To do this I can use theEXTRACT()command. TheEXTRACT()command is a PostgreSQL date/time function that allows you to extract various date/time elements. For our column, PostgreSQL has the specification DOW (day-of-week) which maps 0 to Sunday through to 6 for Saturday.PostgreSQL code:--extract day-of-week from date column and cast the output to an int
SELECT *,
EXTRACT(DOW FROM time)::int AS day_of_week
FROM energy_view ewResults:typetimeusageunitscostnotesday_of_weekElectric usage2016-10-22 00:00:00.0000.01kWh0.006Electric usage2016-10-22 00:15:00.0000.01kWh0.006Electric usage2016-10-22 00:30:00.0000.01kWh0.006Electric usage2016-10-22 00:45:00.0000.01kWh0.006Electric usage2016-10-22 01:00:00.0000.01kWh0.006Electric usage2016-10-22 01:15:00.0000.01kWh0.006Python code:energy_df['day_of_week'] = energy_df['time'].dt.dayofweekAdditionally, we may want to add another column that specifies if a day occurs over a weekend or weekday. I will do this by creating a boolean column, wheretruerepresents a weekend, andfalserepresents a weekday. To do this, I will apply aCASEstatement. With this command I can specify “when-then” statements (similar to “if-then” statements in coding) where I can sayWHENaday_of_weekvalue isINthe set (0,6)THENthe output should betrue,ELSEthe value should befalse.PostgreSQL code:SELECT type, time, usage, units, cost,
EXTRACT(DOW FROM time)::int AS day_of_week, 
--use the case statement to make a column true when records fall on a weekend aka 0 and 6
CASE WHEN (EXTRACT(DOW FROM time)::int) IN (0,6) then true
	ELSE false
END AS is_weekend
FROM energy_view ewResults:typetimeusageunitscostday_of_weekis_weekendElectric usage2016-10-22 00:00:00.0000.01kWh0.006trueElectric usage2016-10-22 00:15:00.0000.01kWh0.006trueElectric usage2016-10-22 00:30:00.0000.01kWh0.006trueElectric usage2016-10-22 00:45:00.0000.01kWh0.006trueElectric usage2016-10-22 01:00:00.0000.01kWh0.006trueFun fact: you can do the same query without aCASEstatement, however it only works for binary columns.--another method to create a binary column
SELECT type, time, usage, units, cost,
EXTRACT(DOW FROM time)::int AS day_of_week, 
EXTRACT(DOW FROM time)::int IN (0,6) AS is_weekend
FROM energy_view ewPython code:Notice that in Python, the weekends are represented by numbers 5 and 6 vs the PostgreSQL weekend values 0 and 6.energy_df['is_weekend'] = np.where(energy_df['day_of_week'].isin([5,6]), 1, 0)
print(energy_df.head(20))And maybe things then start getting real crazy, maybe you want to add more parameters!Let’s consider holidays. Now you may be asking “Why in the world would we do that?!”, but often people have time off during some of the holidays within the US. Since this individual lives within the US, they likely have at leastsomeof the holidays off whether they are the day of OR a federal holiday. Where there are days off, there could be a difference in energy usage. To help guide my analysis, I want to include the identification of holidays. To do this, I’m going to create another boolean column that identifies when a federal holiday occurs.To do this, I am going to use TimescaleDB’stime_bucket()function. Thetime_bucket()function is one of the functions I discussed in detail within myprevious post. Essentially, I need to use this function to make sure all time values within a single day get accounted for. Without using thetime_bucket()function, I would only see changes to the row associated with the 12am time period.PostgreSQL code:After I create a holiday table, I can then use the data from it within my query. I also decided to use the non-case syntax for this query. Note that you can use either!--create table for the holidays
CREATE TABLE holidays (
date date)

--insert the holidays into table
INSERT INTO holidays 
VALUES ('2016-11-11'), 
('2016-11-24'), 
('2016-12-24'), 
('2016-12-25'), 
('2016-12-26'), 
('2017-01-01'),  
('2017-01-02'), 
('2017-01-16'), 
('2017-02-20'), 
('2017-05-29'), 
('2017-07-04'), 
('2017-09-04'), 
('2017-10-9'), 
('2017-11-10'), 
('2017-11-23'), 
('2017-11-24'), 
('2017-12-24'), 
('2017-12-25'), 
('2018-01-01'), 
('2018-01-15'), 
('2018-02-19'), 
('2018-05-28'), 
('2018-07-4'), 
('2018-09-03'), 
('2018-10-8')

SELECT type, time, usage, units, cost,
EXTRACT(DOW FROM time)::int AS day_of_week, 
EXTRACT(DOW FROM time)::int IN (0,6) AS is_weekend,
-- I can then select the data from the holidays table directly within my IN statement
time_bucket('1 day', time) IN (SELECT date FROM holidays) AS is_holiday
FROM energy_view ewResults:typetimeusageunitscostday_of_weekis_weekendis_holidayElectric usage2016-10-22 00:00:00.0000.01kWh0.006truefalseElectric usage2016-10-22 00:15:00.0000.01kWh0.006truefalseElectric usage2016-10-22 00:30:00.0000.01kWh0.006truefalseElectric usage2016-10-22 00:45:00.0000.01kWh0.006truefalseElectric usage2016-10-22 01:00:00.0000.01kWh0.006truefalseElectric usage2016-10-22 01:15:00.0000.01kWh0.006truefalsePython code:holidays = ['2016-11-11', '2016-11-24', '2016-12-24', '2016-12-25', '2016-12-26', '2017-01-01',  '2017-01-02', '2017-01-16', '2017-02-20', '2017-05-29', '2017-07-04', '2017-09-04', '2017-10-9', '2017-11-10', '2017-11-23', '2017-11-24', '2017-12-24', '2017-12-25', '2018-01-01', '2018-01-15', '2018-02-19', '2018-05-28', '2018-07-4', '2018-09-03', '2018-10-8']
energy_df['is_holiday'] = np.where(energy_df['day_of_week'].isin(holidays), 1, 0)
print(energy_df.head(20))At this point, I’m going to save this expanded table into anotherVIEWso that I can call the data without writing out the query.PostgreSQL code:--create another view with the data from our first round of cleaning
CREATE VIEW energy_view_exp AS
SELECT type, time, usage, units, cost,
EXTRACT(DOW FROM time)::int AS day_of_week, 
EXTRACT(DOW FROM time)::int IN (0,6) AS is_weekend,
time_bucket('1 day', time) IN (select date from holidays) AS is_holiday
FROM energy_view ewYou may be asking, “Why did you create these as boolean columns??”, a very fair question! You see, I may want to use these columns for filtering during analysis, something I commonly do during my own analysis process. In PostgreSQL, when you use boolean columns you can filter things super easily. For example, say that I want to use my table query so far and show only the data that occurs over the weekendANDa holiday. I can do this simply by adding in aWHEREstatement along with the specified columns.PostgreSQL code:--if you use binary columns, then you can filter with a simple WHERE statement
SELECT *
FROM energy_view_exp
WHERE is_weekend AND is_holidayResults:typetimeusageunitscostday_of_weekis_weekendis_holidayElectric usage2016-12-24 00:00:00.0000.34kWh0.066truetrueElectric usage2016-12-24 00:15:00.0000.34kWh0.066truetrueElectric usage2016-12-24 00:30:00.0000.34kWh0.066truetrueElectric usage2016-12-24 00:45:00.0000.34kWh0.066truetrueElectric usage2016-12-24 01:00:00.0000.34kWh0.066truetrueElectric usage2016-12-24 01:15:00.0000.34kWh0.066truetruePython code:print(energy_df[(energy_df['is_weekend']==1) & (energy_df['is_holiday']==1)].head(10))Adding data to a hypertableNow that I have new columns ready to go and I know how I would like my table to be structured, I can create a new hypertable and insert my cleaned data. In my own analysis with this data set, I may have done the cleaning up to this point BEFORE evaluating my data so that I can get a more meaningful evaluation step in analysis. What’s great is that you can use any of these techniques for general cleaning, whether that is before or after evaluation.PostgreSQL:CREATE TABLE energy_usage (
type text,
time timestamptz,
usage float4,
units text,
cost float4,
day_of_week int,
is_weekend bool,
is_holiday bool,
) 

--command to create a hypertable
SELECT create_hypertable('energy_usage', 'time')

INSERT INTO energy_usage 
SELECT *
FROM energy_view_expNote that if you had data continually coming in you could create a script within your database that automatically makes these changes when importing your data. That way you can have cleaned data ready to go in your database rather than processing and cleaning the data in your scripts every time you want to perform analysis.We will discuss this in detail in my next post, so make sure to stay tuned in if you want to know how to create scripts and keep data automatically updated!Renaming valuesAnother valuable technique for cleaning data is being able to rename various items or remap categorical values. The importance of this skill is amplified by thepopularity of this Python data analysis question on StackOverflow. The question states “How do I change a single index value in a pandas dataframe?”. Since PostgreSQL and TimescaleDB use relational table structures, renaming unique values can be fairly simple.When renaming specific index values within a table, you can do this “on the fly” by using PostgreSQL’sCASEstatement within theSELECTquery. Let’s say I don’t like Sunday being represented by a 0 in theday_of_weekcolumn, but would prefer it to be a 7. I can do this with the following query.PostgreSQL code:SELECT type, time, usage, cost, is_weekend,
-- you can use case to recode column values 
CASE WHEN day_of_week = 0 THEN 7
ELSE day_of_week 
END
FROM energy_usagePython code:Caveat, this code would make Monday = 7 because the python DOW function has Monday set to 0 and Sunday set to 6. But this is how you would update one value within a column. Likely you would not want to do this exact action, I just wanted to show the python equivalent for reference.energy_df.day_of_week[energy_df['day_of_week']==0] = 7
print(energy_df.head(250))Now, let’s say that I wanted to actually use the names of the days of the week instead of showing numeric values? For this example, I actually want to ditch theCASEstatement and create a mapping table. When you need to change various values, it will likely be more efficient to create a mapping table and then join to this table using theJOINcommand.PostgreSQL:--first I need to create the table
CREATE TABLE day_of_week_mapping (
day_of_week_int int,
day_of_week_name text
)

--then I want to add data to my table
INSERT INTO day_of_week_mapping
VALUES (0, 'Sunday'),
(1, 'Monday'),
(2, 'Tuesday'),
(3, 'Wednesday'),
(4, 'Thursday'),
(5, 'Friday'),
(6, 'Saturday')

--then I can join this table to my cleaning table to remap the days of the week
SElECT type, time, usage, units, cost, dowm.day_of_week_name, is_weekend
FROM energy_usage eu
LEFT JOIN day_of_week_mapping dowm ON dowm.day_of_week_int = eu.day_of_weekResults:typetimeusageunitscostday_of_week_nameweekendElectric usage2018-07-22 00:45:00.0000.1kWh0.03SundaytrueElectric usage2018-07-22 00:30:00.0000.1kWh0.03SundaytrueElectric usage2018-07-22 00:15:00.0000.1kWh0.03SundaytrueElectric usage2018-07-22 00:00:00.0000.1kWh0.03SundaytrueElectric usage2018-02-11 23:00:00.0000.04kWh0.01SundaytruePython:In this case, python has similar mapping functions.energy_df['day_of_week_name'] = energy_df['day_of_week'].map({0 : 'Sunday', 1 : 'Monday', 2: 'Tuesday', 3: 'Wednesday', 4: 'Thursday', 5: 'Friday', 6: 'Saturday'})
print(energy_df.head(20))Hopefully, one of these techniques will be useful for you as you approach data renaming!Additionally, remember that if you would like to change the name of a column in your table, it is truly as easy asAS(I couldn’t not use such a ridiculous statement 😂). When you use theSELECTstatement, you can rename your columns like so,PostgreSQL code:SELECT type AS usage_type,
time as time_stamp,
usage,
units, 
cost AS dollar_amount
FROM energy_view_exp
LIMIT 20;Results:usage_typetime_stampusageunitsdollar_amountElectric usage2016-10-22 00:00:00.0000.01kWh0.00Electric usage2016-10-22 00:15:00.0000.01kWh0.00Electric usage2016-10-22 00:30:00.0000.01kWh0.00Electric usage2016-10-22 00:45:00.0000.01kWh0.00Python code:Comparatively, renaming columns in Python can be a huge pain. This is an area where SQL is not only faster, but also just more elegant in its code.energy_df.rename(columns={'type':'usage_type', 'time':'time_stamp', 'cost':'dollar_amount'}, inplace=True)
print(energy_df[['usage_type','time_stamp','usage','units','dollar_amount']].head(20))Fill in missing dataAnother common problem in the data cleaning process is having missing data. For the dataset we are using, there are no obviously missing data points, however, it is very possible that with evaluation, we could find missing hourly data from a power outage or some other phenomenon. This is where the gap-filling functions TimescaleDB offers could come in handy. When using algorithms, missing data can often have significant negative impacts on the accuracy or dependability of the model. Sometimes, you can navigate this problem by filling in missing data with reasonable estimates and TimescaleDB actually has built-in functions to help you do this.For example, let’s say that you are modeling the energy usage over individual days of the week and a handful of days have missing energy data due to a power outage or an issue with the sensor. We could remove the data, or try to fill in the missing values with reasonable estimations. For today, let’s assume that the model I want to use would benefit more from filling in the missing values.As an example, I created some data. I called this tableenergy_dataand it is missing bothtimeandenergyreadings for the timestamps between 7:45am and 11:30am.timeenergy2021-01-01 07:00:00.00002021-01-01 07:15:00.0000.12021-01-01 07:30:00.0000.12021-01-01 07:45:00.0000.22021-01-01 11:30:00.0000.042021-01-01 11:45:00.0000.042021-01-01 12:00:00.0000.032021-01-01 12:15:00.0000.022021-01-01 12:30:00.0000.032021-01-01 12:45:00.0000.022021-01-01 13:00:00.0000.03I can use TimescaleDB’sgapfilling hyperfunctionsto fill in these missing values. Theinterpolate()function is another one of TimescaleDB’s hyperfunctions and it creates data points that follow a linear approximation given the data points before and after the missing range of data. Alternatively, you could use thelocf()hyperfunction which carries the last recorded value forward to fill in the gap (note that locf stands for last-one-carried-forward). Both of these functions must be used in conjunction with thetime_bucket_gapfill()function.PostgreSQL code:SELECT
--here I specified that the data should increment by 15 mins
  time_bucket_gapfill('15 min', time) AS timestamp,
  interpolate(avg(energy)),
  locf(avg(energy))
FROM energy_data
--to use gapfill, you will have to take out any time data associated with null values. You can do this using the IS NOT NULL statement
WHERE energy IS NOT NULL AND time > '2021-01-01 07:00:00.000' AND time < '2021-01-01 13:00:00.000'
GROUP BY timestamp
ORDER BY timestamp;Results:timestampinterpolatelocf2021-01-01 07:00:00.0000.10.100000000000000000002021-01-01 07:30:00.0000.150.150000000000000000002021-01-01 08:00:00.0000.136250.150000000000000000002021-01-01 08:30:00.0000.12250.150000000000000000002021-01-01 09:00:00.0000.108750.150000000000000000002021-01-01 09:30:00.0000.0950.150000000000000000002021-01-01 10:00:00.0000.081250.150000000000000000002021-01-01 10:30:00.0000.06750.150000000000000000002021-01-01 11:00:00.0000.053750.150000000000000000002021-01-01 11:30:00.0000.040.040000000000000000002021-01-01 12:00:00.0000.0250.025000000000000000002021-01-01 12:30:00.0000.0250.02500000000000000000Python code:energy_test_df['time'] = pd.to_datetime(energy_test_df['time'])
energy_test_df_locf = energy_test_df.set_index('time').resample('15 min').fillna(method='ffill').reset_index()
energy_test_df = energy_test_df.set_index('time').resample('15 min').interpolate().reset_index()
energy_test_df['locf'] = energy_test_df_locf['energy']
print(energy_test_df)Bonus:The following query is how I could ignore the missing data. I wanted to include this to show you just how easy it can be to exclude null data. Alternatively, I could use aWHEREclause to specify the times which I could like to ignore (the second query).SELECT * 
FROM energy_data 
WHERE energy IS NOT NULL

SELECT * 
FROM energy_data
WHERE time <= '2021-01-01 07:45:00.000' OR time >= '2021-01-01 11:30:00.000'Wrap UpAfter reading through these various cleaning techniques, I hope you feel more comfortable with exploring some of the possibilities that PostgreSQL and TimescaleDB provide. By cleaning data directly within my database, I am able to perform a lot of my cleaning tasks a single time rather than repetitively within a script, thus saving me time in the long run. If you are looking to save time and effort while cleaning your data for analysis, definitely consider using PostgreSQL and TimescaleDB.In my next posts, I will go over techniques on how to transform data using PostgreSQL and TimescaleDB. I'll then take everything we've learned together to benchmark data munging tasks in PostgreSQL and TimescaleDB vs. Python and pandas. The final blog post will walk you through the full process on a real dataset by conducting a deep-dive into data analysis with TimescaleDB (for data munging) and Python (for modeling and visualizations).If you have questions about TimescaleDB, time-series data, or any of the functionality mentioned above, join ourcommunity Slack, where you'll find an active community of time-series enthusiasts and various Timescale team members (including me!).If you’re ready to see the power of TimescaleDB and PostgreSQL right away, you cansign up for a free 30-day trialor install TimescaleDB andmanage it on your current PostgreSQL instances. We also have a bunch ofgreat tutorialsto help get you started.Until next time!Functionality Glossary:Adding columns togetherTO_NUMBER()VIEWWHEREEXTRACT()CASEtime_bucket()JOINASCREATE TABLEcreate_hypertable()INSERT INTOtime_bucket_gapfill()Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/postgresql-vs-python-for-data-cleaning-a-guide/
2021-12-22T20:24:08.000Z,Timescale flies when you’re having fun: 2021 in review,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Welcome to the last blog post of the year!It has become a Timescale tradition to celebrate#12DaysofTimescale, where we share our favorite content pieces published the previous year. But 2021 was such an awesome year for us in Team Timescale, that we didn’t want to end it there.Today, we are celebrating the accomplishments that made us proud in 2021.In 2021, Timescale delivered substantial improvements to our category-defining relational database for time-series. Early in our history, Timescale made the decision to build on PostgreSQL, the world’s fastest-growing database. Today, that foundation enables us to move quickly and deliver the functionality and dependability that developers demand when building data-intensive applications.We shipped Timescale, a cloud database that makes it incredibly easy for developers to build and deploy data-intensive applications. We advanced the state-of-the-art in SQL with new hyperfunctions and function pipelines, tools that significantly improve developer productivity by making it easier to manipulate and analyze time-series data using the SQL you know and love. We made significant investments in the observability business by embracing OpenTelemetry and making Promscale your unified store for metrics, logs, and traces.4,000,000 TimescaleDB instances are active today.Developers count on us every day. We are proud of our contributions to the developer community and look forward to next year when we can continue to delight our customers as they build and deploy data-intensive applications.Read on for a summary of everything we’ve shipped this year.If you’re new to Timescale, check out Timescale;you can create a free account to get started(100% free for 30 days and no credit card required).Once you are using Timescale,join the Timescale Communityand ask us any questions about time-series data, TimescaleDB, PostgreSQL, and more. And, for those who share our mission of serving developers worldwide 🌏 and want to join our fully remote, global team,we are hiring broadly across many roles!Timescale ⛅In October, welaunched the new and improved Timescale. Timescale provides a balance between a familiar developer platform and a flexible database for modern applications while retaining the ease and scalability of modern cloud services.Timescale represents our vision for a modern data platform that combines the ease of use expected from a cloud service with the flexibility that developers need. We call this the database cloud. The database cloud provides developers with a familiar interface so that they can bring their existing skills to solve the complex problems of today, be it in financial services, web-scale infrastructure, IoT, or more. The database cloud gives developers the ability to dig deep into the internals of the database and truly understand how their data is managed. This year, we shipped many new features for Timescale, including:Multi-region supportOne-click support for multi-node servicesVPC peeringStorage autoscalingwith zero downtime and configurable limitsExplorerdashboard for an easy, visual in-console interfaceAutomated, zero-downtime upgrades duringmaintenance windowsPoint-in-time recovery via automated, continuous incremental backupsGuided demowith pre-loaded data for first-time usersPDF invoicesAdditional emails for billingOne-click database forkingNew features in TimescaleDB 🐯At Timescale, we focus every day on delighting our customers with powerful and easy-to-use features. In 2021, we delivered 13 TimescaleDB releases, including:Support for PG14. We want TimescaleDB users to benefit from the latest improvements introduced by the Postgres community, sowe included support for PostgreSQL 14 only one month after its release.Schema modificationsandinserts in compressed hypertables.In TimescaleDB 2.3 and above, you can directly insert data into compressed hypertables (vs decompressing, inserting, and then re-compressing chunks manually).Drastic performance optimizations through Skip Scan. We implemented a technique that gave users 26x-8000x fasterDISTINCTqueries, not only in TimescaleDB hypertables but in their regular PostgreSQL tables as well.Experimental schema. In TimescaleDB 2.4 we introduced our experimentalschema, where we develop and ship features at an even faster pace than we normally do - to later “graduate” these experimental features once they reach full maturity for normal production usage.More flexibility intime_bucket()throughtime_bucket_ng(). A good example of the use of the experimental schema istime_bucket_ng(), which expands the populartime_bucket()to also support buckets by months, years, andtimezones.Apart from these major feature releases, we improved the functionality of distributed hypertables, adding support for advanced TimescaleDB features likecompression policies,continuous aggregates,triggers,gap-fill, andSkip Scan. We also added advanced functionality in regards to multi-node operations, like the ability tomoveandcopy chunksbetween data nodes,distributed restore points for backup/restore, and managing distributed objects and DDL.And… We closed728 issuesin Github! Thank you so much to everyone who reported bugs and who helped us improve TimescaleDB: this wouldn’t be possible without you! We also love hearing which features you would like to see next (closing those issues feels especially satisfying).Please, keep sharing your suggestions and feedback with us!In July,we launched TimescaleDB hyperfunctions, a series of SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in PostgreSQL with fewer lines of code. You can use hyperfunctions to calculate percentile approximations of data, compute time-weighted averages, downsample and smooth data, and perform fasterCOUNT DISTINCTqueries using approximations. Moreover, hyperfunctions are “easy” to use: you call a hyperfunction using the same SQL syntax you know and love.We’ve already delivered all these hyperfunctions:approximate_row_countfirstlasthistogramtime_buckettime_bucket_nghyperloglogrollupdistinct_countstderrorstats_aggrollingrollupaverage / average_y / average_xcorr (correlation coefficient)covariancedetermination_coeff (R squared)interceptkurtosis / kurtosis_y / kurtosis_xnum_valsskewness / skewness_y / skewness_xstddev / stddev_y / stddev_xslopesum/ sum_y / sum_xvariance / variance_y / variance_xx_intercepttime_bucket_gapfilllocfinterpolatepercentile_aggapprox_percentileapprox_percentile_rankrollupmax_valmeanerrormin_valnum_valscounter_agg (point form)rollupcorrcounter_zero_timedeltaextrapolated_deltaextrapolated_rateideltainterceptiratenum_changesnum_elementsnum_resetsrateslopetime_deltawith_boundstime_weightrollupaverageasaplttbAnd in October,we launched function pipelines, a new capability that introduces functional programming concepts inside PostgreSQL (and SQL) using custom operators. Function pipelines radically improve the developer ergonomics of analyzing data in PostgreSQL and SQL, by applying principles from functional programming and popular tools like Python’s Pandas and PromQL.SELECT device id, 
	sum(abs_delta) as volatility
FROM (
	SELECT device_id, 
		abs(val - lag(val) OVER (PARTITION BY device_id ORDER BY ts))
        	as abs_delta 
	FROM measurements
	WHERE ts >= now() - '1 day'::interval) calc_delta
GROUP BY device_id;Pop quiz: What does this query do?Even if you are a SQL expert, queries like this can be quite difficult to read - and even harder to express. Complex data analysis in SQL can be hard.Function pipelines let you express that same query like this:SELECT device_id, 
	timevector(ts, val) -> sort() -> delta() -> abs() -> sum() 
    		as volatility
FROM measurements
WHERE ts >= now() - '1 day'::interval
GROUP BY device_id;Function pipelines: Building functional programming into PostgreSQL using custom operatorsOne Promscale for all 🔍In October,we made major improvements to Promscale. Promscale started as a data store for Prometheus metrics, but quickly matured in 2021 to become a unified backend for observability data - supporting both Prometheus metrics and OpenTelemetry traces. And this is only the beginning (stay tuned!)Apart from the support for OpenTelemetry tracing in Promscale, we released many tools and improvements related to Promscale and observability, including:Tobs, a tool that allows you to deploy a full observability stack for Kubernetes in one command. The stack includes Promscale, TimescaleDB, Prometheus, OpenTelemetry Operator, Jaeger, and Grafana (including pre-built dashboards for Kubernetes)Prom-migrator, a universal migration tool for Prometheus dataMulti-tenancy in PromscaleSupport for OpenMetrics exemplars, continuous aggregates, Prometheus metadata, high availability setups, PG14, text + JSON ingest API, and multi-nodeInnumerable bug and performance fixesThank youAll of us at Timescale are so proud to work for you every day ❤️ We hope the year ahead brings you great joy and prosperity. Please, stay safe during this holiday season!We wish you happy holidays and a happy New Year.On to 2022!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescale-flies-when-youre-having-fun-2021-in-review/
2021-09-09T15:32:12.000Z,Speeding up data analysis with TimescaleDB and PostgreSQL,"Time-series datais everywhere, and it drives decision-making in every industry. Time-series data collectively represents how a system, process, or behavior changes over time. Understanding these changes helps us to solve complex problems across numerous industries, includingobservability,financial services,Internet of Things, and evenprofessional football.Depending on the type of application they’re building, developers end up collecting millions of rows of time-series data (and sometimes millions of rows of data every day or even every hour!). Making sense of this high-volume, high-fidelity data takes a particular set of data analysis skills that aren’t often exercised as part of the classic developer skillset. To perform time-series analysis that goes beyond basic questions, developers and data analysts need specialized tools, and astime-series data grows in prominence, theefficiencyof these tools becomes even more important.Often, data analysts’ work can be boiled down toevaluating,cleaning,transforming, andmodelingdata. In my experience, I’ve found these actions are necessary for me to gain understanding from data, and I will refer to this as the “data analysis life cycle” throughout this post.Data analysis lifecycleExcel, R, and Python are arguably some of the most commonly used data analysis tools, and, while they are all fantastic tools, they may not be suited for every job. Speaking from experience, these tools can be especially inefficient for “data munging” at the early stages of the lifecycle; specifically, theevaluating data,cleaning data, andtransforming datasteps involved in pre-modeling work.As I’ve worked with larger and more complex datasets, I’ve come to believe that databases built for specific types of data - such as time-series data - are more effective for data analysis.For background,TimescaleDBis arelationaldatabase for time-series data. If your analysis is based on time-series datasets, TimescaleDB can be a great choice not only for its scalability and dependability but also for its relational nature. Because TimescaleDB is packaged as an extension to PostgreSQL, you’ll be able to look at your time-series data alongside your relational data and get even more insight. (I recognize that as a Developer Advocate at Timescale, I might be alittlebiased 😊…)In this blog series, I will discuss each of the three data munging steps in the analysis lifecycle in-depth and demonstrate how to use TimescaleDB as a powerful tool for your data analysis.In this introductory post, I'll explore a few of the common frustrations that I experienced with popular data analysis tools, and from there, dive into how I’ve used TimescaleDB to help alleviate each of those pain points.In future posts we'll look at:How TimescaleDB data analysis functionality can replace work commonly performed in Python and pandasHow TimescaleDB vs. Python and pandas compare (benchmarking a standard data analysis workflow)How to use TimescaleDB to conduct an end-to-end, deep-dive data analysis, using real yellow taxi cab data from theNew York City Taxi and Limousine Commission(NYC TLC).If you are interested in trying out TimescaleDB and PostgreSQL functionality right away,sign up for a free 30-day trialorinstall and manage it on your instances. (You can also learn more byfollowing one of our many tutorials.)Common data analysis tools and “the problem”As we’ve discussed, the three most popular tools used for data analysis are Excel, R, and Python. While they are great tools in their own right, they are not optimized to efficiently perform every step in the analysis process.In particular, most data scientists (including myself!) struggle with similar issues as the amount of data grows or the same analysis needs to be redone month after month.Some of these struggles include:Data storage and access: Where is the best place to store and maintain my data for analysis?Data size and its influence on the analysis: How can I improve efficiency for data munging tasks, especially as data scales?Script storage and accessibility: What can I do to improve data munging script storage and maintenance?Easily utilizing new technologies: How could I set up my data analysis toolchain to allow for easy transitions to new technologies?So buckle in, keep your arms and legs in the vehicle at all times, and let’s start looking at these problems!Data analysis issue #1: storing and accessing dataTo do data analysis, you need access to… data.via GIPHYManaging where that data lives, and how easily you can access it is the preliminary (and often most important) step in the analysis journey. Every time I begin a new data analysis project, this is often where I run into my first dilemma. Regardless of the original data source, I always ask “where is the best place to store and maintain the data as I start working through the data munging process?”Although it's becoming more common for data analysts to use databases for storing and querying data, it's still not ubiquitous. Too often, raw data is provided in a stream of CSV files or APIs that produce JSON. While this may be manageable for smaller projects, it can quickly become overwhelming to maintain and difficult to manage from project to project.For example, let’s consider how we might use Python as our data analysis tool of choice.While using Python for data analysis, I have the option of ingesting data through files/APIs OR a database connection.If I used files or APIs for querying data during analysis, I often faced questions like:Where are the files located? What happens if the URL or parameters change for an API?What happens if duplicate files are made? And what if updates are made to one file, and not the other?How do I best share these files with colleagues?What happens if multiple files depend on one another?How do I prevent incorrect data from being added to the wrong column of a CSV? (ie. a decimal where a string should be)What about very large files? What is the ingestion rate for a 10MB, 100MB, 1GB, 1TB sized file?After running into these initial problems project after project, I knew there had to be a better solution.I knew that I needed a single source of truth for my data – and it started to become clear that a specialized SQL database might be my answer!Now, let’s consider if I were to connect to TimescaleDB.By importing my time-series data into TimescaleDB, I can create one source of truth for all of my data. As a result, collaborating with others becomes as simple as sharing access to the database. Any modifications to the data munging process within the database means that all users have access to the same changes at the same time, opposed to parsing through CSV files to verify I have the right version.Additionally, databases can typically handle much larger data loads than a script written in Python or R. TimescaleDB was built to house, maintain, and query terabytes of data efficiently and cost-effectively (both computationally speaking AND for your wallet). With features likecontinuous aggregatesand native columnarcompression, storing and analyzing years of time-series data became efficient while still being easily accessible.In short, managing data over time, especially when it comes from different sources, can be a nightmare to maintain and access efficiently. But, it doesn’t have to be.Data analysis issue #2: maximizing analysis speed and computation efficiency (the bigger the dataset, the bigger the problem)Excel, R, and Python are all capable of performing the first three steps of the data analysis “lifecycle”: evaluating, cleaning, and transforming data. However, these technologies are not generally optimized for speed or computational efficiency during the process.In numerous projects over the years, I’ve found that as the size of my dataset increased, the process of importing, cleaning, and transforming it became more difficult, time-consuming, and, in some cases impossible. For Python and R, parsing through large amounts of data seemed to take forever, and Excel would simply crash once hitting millions of rows.Things becameespeciallydifficult when I needed to create additional tables for things like aggregates or data transformations: some lines of code could take seconds or, in extreme cases, minutes to run depending on the size of the data, the computer I was using, or the complexity of the analysis.While seconds or minutes may notseemlike a lot, it adds up and amounts to hours or days of lost productivity when you’re performing analysis that needs to be run hundreds or thousands of times a month!To illustrate, let’s look at a Python example once again.Say I was working withthis IoT data set taken from Kaggle. The set contains two tables, one specifying energy consumption for a single home in Houston Texas, and the other documenting weather conditions.To run through analysis with Python, the first steps in my analysis would be to pull in the data and observe it.When using Python to do this, I would run code like this 👇import psycopg2
import pandas as pd
import configparser


## use config file for database connection information
config = configparser.ConfigParser()
config.read('env.ini')

## establish conntection
conn = psycopg2.connect(database=config.get('USERINFO', 'DB_NAME'), 
                        host=config.get('USERINFO', 'HOST'), 
                        user=config.get('USERINFO', 'USER'), 
                        password=config.get('USERINFO', 'PASS'), 
                        port=config.get('USERINFO', 'PORT'))

## define the queries for selecting data out of our database                        
query_weather = 'select * from weather'
query_power = 'select * from power_usage'

## create cursor to extract data and place it into a DataFrame
cursor = conn.cursor()
cursor.execute(query_weather)
weather_data = cursor.fetchall()
cursor.execute(query_power)
power_data = cursor.fetchall()
## you will have to manually set the column names for the data frame
weather_df = pd.DataFrame(weather_data, columns=['date','day','temp_max','temp_avg','temp_min','dew_max','dew_avg','dew_min','hum_max','hum_avg','hum_min','wind_max','wind_avg','wind_min','press_max','press_avg','press_min','precipit','day_of_week'])
power_df = pd.DataFrame(power_data, columns=['startdate', 'value_kwh', 'day_of_week', 'notes'])
cursor.close()

print(weather_df.head(20))
print(power_df.head(20))Altogether, this code took 2.718 seconds to run using my2019 MacBook Pro laptop with 32GB memory.But, what about if I run this equivalent script with SQL in the database?select * from weather
select * from power_usagestartdatevalue_kwhday_of_weeknotes2016-01-06 01:00:0012weekday2016-01-06 02:00:0012weekday2016-01-06 03:00:0012weekday2016-01-06 04:00:0012weekday2016-01-06 05:00:0002weekday2016-01-06 06:00:0002weekday2016-01-06 07:00:0002weekday2016-01-06 08:00:0002weekday2016-01-06 09:00:0002weekday2016-01-06 10:00:0002weekday2016-01-06 11:00:0012weekday2016-01-06 12:00:0002weekday2016-01-06 13:00:0002weekday2016-01-06 14:00:0002weekday2016-01-06 15:00:0002weekday2016-01-06 16:00:0012weekday2016-01-06 17:00:0042weekdayThis query only took 0.342 seconds to run, almost 8x faster when compared to the Python script.This time difference makes a lot of sense when we consider that Python must connect to a database, then run the SQL query, then parse the retrieved data, and then import it into a DataFrame. While almost three seconds is fast, this extra time for processing adds up as the script becomes more complicated and more data munging tasks are added.Pulling in the data and observing it is only the beginning of my analysis! What happens when I need to perform a transforming task, like aggregating the data?For this dataset, when we look at thepower_usagetable - as seen above - kWh readings are recorded every hour. If I want to do daily analysis, I have to aggregate the hourly data into “day buckets”.If I used Python for this aggregation, I could use something like 👇# sum power usage by day, bucket by day
## create column for the day 
day_col = pd.to_datetime(power_df['startdate']).dt.strftime('%Y-%m-%d')
power_df.insert(0, 'date_day', day_col)
agg_power = power_df.groupby('date_day').agg({'value_kwh' : 'sum', 'day_of_week' : 'unique', 'notes' : 'unique' })
print(agg_power)...which takes 0.49 seconds to run (this does not include the time for importing our data).Alternatively, with the TimescaleDBtime_bucket()function, I could do this aggregation directly in the database using the following query 👇select 
	time_bucket(interval '1 day', startdate ) as day,
	sum(value_kwh),
	day_of_week,
	notes
from power_usage pu 
group by day, day_of_week, notes
order by daydaysumday_of_weeknotes2016-01-06 00:00:00272weekday2016-01-07 00:00:00423weekday2016-01-08 00:00:00514weekday2016-01-09 00:00:00505weekend2016-01-10 00:00:00456weekend2016-01-11 00:00:00220weekday2016-01-12 00:00:00121weekday2016-02-06 00:00:00325weekend2016-02-07 00:00:00626weekend2016-02-08 00:00:00480weekday2016-02-09 00:00:00231weekday2016-02-10 00:00:00242weekday...which only takes 0.087 seconds and is over 5x faster than the Python script.You can start to see a pattern here.As mentioned above,TimescaleDB was created to efficiently query and store time-series data. But simply querying data only scratches the surface of the possibilities TimescaleDB and PostgreSQL functionality provides.TimescaleDB and PostgreSQL offer a wide range of tools and functionality that can replace the need for additional tools to evaluate, clean, and transform your data. Some of the TimescaleDB functionality includes continuous aggregates, compression, andhyperfunctions; all of which allow you to do nearly all data munging tasks directly within the database.When I performed the evaluating, cleaning, and transforming steps of my analysis directly within TimescaleDB, I cut out the need to use additional tools - like Excel, R, or Python - for data munging tasks. I could pull cleaned and transformed data, ready for modeling, directly into Excel, R, or Python.Data analysis issue #3: storing and maintaining scripts for data analysisAnother potential downside of exclusively using Excel, R, or Python for the entire data analysis workflow, is that all of the logic for analyzing the data is contained within a script file. Similar to the issues of having many different data sources, maintaining script files can be inconvenient and messy.Some common issues that I - and many data analysts - run into include:Losing filesUnintentionally creating duplicate filesChanging or updating some files but not othersNeeding to write and run scripts to access transformed data (see below example)Spending time re-running scripts whenever new raw data is added (see below example)While you can use a code repository to overcome some of these issues, it will not fix the last two.Let’s consider our Python scenario again.Say that I used a Python script exclusively for all my data analysis tasks. What happens if I need to export my transformed data to use in a report on energy consumption in Texas?Likely, I would have to add some code within the script to allow for exporting the data and then run the script again to actually export it. Depending on the content of the script and how long it takes to transform the data, this could be pretty inconvenient and inefficient.What if I also just got a bunch of new energy usage and weather data? For me to incorporate this new raw data into existing visualizations or reports, I would need to run the script again and make sure that all of my data munging tasks run as expected.Database functions, like continuous aggregates and materialized views, can create transformed data that can be stored and queried directly from your database without running a script. Additionally, I can create policies for continuous aggregates to regularly keep this transformed data up-to-date any time raw data is modified. Because of these policies, I wouldn't have to worry about running scripts to re-transform data for use, making access to updated data efficient. With TimescaleDB, many of the data munging tasks in the analysis lifecycle that you would normally do within your scripts can be accomplished using built-in TimescaleDB and PostgreSQL functionality.Data analysis issue #4: easily utilizing new or additional technologiesFinally, the last step in the data analysis lifecycle: modeling. If I wanted to use a new tool or technology to create a visualization, it was difficult to easily take my transformed data and use it for modeling or visualizations elsewhere.Python, R, and Excel are all pretty great for their visualization and modeling capabilities. However, what happens when your company or team wants to adopt anewtool?In my experience, this often means either adding on another step to the analysis process, or rediscovering how to perform the evaluating, cleaning, and transforming steps within the new technology.For example, in one of my previous jobs, I was asked to convert a portion of my analysis into Power BI for business analytics purposes. Some of the visualizations my stakeholders wanted required me to access transformed data from my Python script. At the time, I had the option to export the data from my Python script or figure out how to transform the data in Power BI directly. Both options were not ideal and were guaranteed to take extra time.When it comes to adopting new visualization or modeling tools, using a database for evaluating, cleaning, and transforming data can again work in your favor. Most visualization tools - such asGrafana,Metabase, orPower BI- allow users to import data from a database directly.Since I can do most of my data munging tasks within TimescaleDB, adding or switching tools - such as using Power BI for dashboard capabilities - becomes as simple as connecting to my database, pulling in the munged data,  and using the new tool for visualizations and modeling.Wrapping upIn summary, Excel, R, and Python are all great tools to use for analysis, but may not be the best tools for every job. Case in point: my struggles with time-series data analysis, especially on big datasets.With TimescaleDB functionality, you can house your data and perform the evaluating, cleaning, and transforming aspects of data analysis, all directly within your database – and solve a lot of common data analysis woes in the process (which I’ve - hopefully! - demonstrated in this post)In the blog posts to come, I’ll explore TimescaleDB and PostgreSQL functionality compared to Python, benchmark TimescaleDB performance vs. Python and pandas for data munging tasks, and conduct a deep-dive into data analysis with TimescaleDB (for data munging) and Python (for modeling and visualizations).If you have questions about TimescaleDB, time-series data, or any of the functionality mentioned above,join ourcommunity Slack, where you'll find an active community of time-series enthusiasts and various Timescale team members (including me!).If you’re ready to see the power of TimescaleDB and PostgreSQL right away, you can sign up fora free 30-day trialorinstall TimescaleDBand manage it on your current PostgreSQL instances. We also have a bunch of greattutorialsto help get you started.Until next time!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/speeding-up-data-analysis/
2021-10-05T13:07:12.000Z,"Announcing the New Timescale, and a New Vision for the Future of Database Services in the Cloud","Timescale provides a balance between a familiar developer platform and a flexible database for modern applications, while retaining the ease and scalability of modern cloud services.All data is time-series data.That was our mantra when we launched TimescaleDB 4.5 years ago. All data has a time dimension, yet most databases only provide a static snapshot of the current state. Instead, by storing data along that time dimension (“time-series data”), we get the dynamic view of what is happening right now, how that is changing, and why that is changing. We get to watch a movie, not just a static image.Similarly, we started this company with the conviction that PostgreSQL is the best foundation for applications. PostgreSQL is proven, versatile, and extensible. It has the fastest growing open-source database community today. It has a broad ecosystem of tooling, connectors, libraries, visualization applications, and more. And most importantly, PostgreSQL isboring: you want your database to be like your Internet, it should be fast, reliable, andjust work.Time has proven that we were correct in these beliefs. Today, Timescale users are pushing the envelope across every industry, including companies like: Akamai, Bosch, Cisco, Comcast, Credit Suisse, DigitalOcean, Electronic Arts, HPE, IBM, Microsoft, Nutanix, NYSE, OpenAI, Rackspace, Schneider Electric, Samsung, Siemens, Uber, Walmart, Warner Music, and many more. The vibrant Timescale community now runs over 3 million active databases every month, enabling them to measure everything that matters across a myriad of use cases, like software applications, industrial equipment, financial markets, blockchain activity, consumer behavior, machine learning models, and climate change, to name but a few.But developers’ preferences continue to evolve. In particular, developers now increasingly turn to managed database services, instead of running their databases in-house. We recognized this shift last year, when we went“all in” on the cloudby making all our previous enterprise features free for the community, and choosing to only monetize via our managed cloud services.Thanks to the success of that move,we raised $40 millionin an oversubscribed round last May,led by Redpoint Ventures(investors in Snowflake, Twilio, Stripe, HashiCorp, Heroku), who called our cloud business “one of the fastest-growing database businesses we have seen in the past 20+ years.”Managed database services are the future, yet we find today’s landscape of managed database services to be lacking. At one extreme, we see database-as-a-service (DBaaS) offerings that still maintain the server metaphor of the self-managed world. These “cloud databases” compete on price and performance, yet the developer experience is often an afterthought. At the other extreme, serverless data platforms optimize for the developer experience, but as a consequence they hide the database and underlying software architecture behind opaque APIs, leading to more developer confusion and vendor lock-in. (And they end up nickel-and-diming developers for every little insert and query.)We believe there’s a better way. The future is serverlessbut notdatabase-less. Developers want a truly easy and worry-free experience but shouldn’t have to blindly trust a black box for their core applications. Developers want a database that is easy to get started, easy to use, and easy to scale so they can focus on their applications. But they also want to understand, diagnose, and even tinker with their database. Thus, the modern database needs to remain familiar and flexible.Today, we are sharing our vision for a modern data platform that combines the ease of use expected from a cloud service with the flexibility that developers need. We call this thedatabase cloud.The database cloud provides developers with a familiar interface so that they can bring their existing skills to solve the complex problems of today, be it in financial services, web-scale infrastructure, IoT, or more. The database cloud gives developers the ability to dig deep into the internals of the database and truly understand how their data is managed.If “black box” services require developers to sacrifice this understanding, then the “transparent box” developers get with the database cloud empowers them.We’re also announcing the new Timescale, a database cloud for relational and time-series workloads, built on PostgreSQL and architected around this new vision. Timescale is not a database that somebody else manages on somebody else’s cloud infrastructure (“a cloud database”) but a full cloud experience built around and for databases (“a database cloud”).These are the first announcements of many this month, our second launch month of the year. This past May, we kicked off our first launch month – an ambitious effort to execute 10+ launches throughout the month (#AlwaysBeLaunching) – andit was a huge success.This month, we are kicking off a second launch month, with another 10+ launches, starting with this blog post today.To learn more about the new Timescale and our new vision for the future of database services in the cloud, please continue.Or to try Timescale today (for free!), pleasesign up here.The database cloud: serverless, but not database-lessAt Timescale, we are dedicated to serving developers worldwide, enabling them to build exceptional data-driven products that measure everything that matters: software applications, industrial equipment, financial markets, blockchain activity, consumer behavior, machine learning models, climate change, and more.At the core of these data-driven products are great databases.TimescaleDB(aka “Postgres for time-series”) is our core product: 100% free, open source (“open core” to be precise, and all on GitHub), and built on Postgres.Developers who use TimescaleDB get the benefit of a purpose-built time-series database plus a classic relational (Postgres) database, all in one, with full SQL (not “SQL-like”) support. And with our scale-out multi-node functionality introduced in TimescaleDB 2.0, TimescaleDB now powers petabyte-scale workloads. A relational database that scales, all for free.A funny thing, though: if databases are doing their job correctly, much like your Internet connection (or your plumbing), you shouldn’t have to think about it. You should be able to leverage skills and knowledge you already have, including query languages you already know (i.e., SQL), and existing tools and applications that just work. You shouldn’t have to worry about reliability, availability, provisioning, backups, resource scaling, performance optimizations, or configuration tuning. Not just today when starting off, but also in the future as your workload scales or use cases grow more complex.The evolution from a self-managed database to a managed database service has been a big step forward for this worry-free experience. Yet DBaaS services that maintain a strong physical server metaphor still offload much of an “easy” and “scaling” burden to developers, who need to endlessly tweak hardware instance and cluster configurations to achieve the scale that modern applications demand. That’s where the serverless paradigm comes in (and why it has taken off). Serverless data platforms, at least in theory, free the developer from thinking about scaling and appear easy to start.Where today’s serverless data platforms fall shortServerless data platforms and APIs are essentially SaaS services. At first they seem deceptively simple. You get a network endpoint you can GET and POST. Everything seems to go fine at first as you onboard onto a new platform intheirpreferred manner.But as you dig deeper, you realize these APIs are not familiar. You soon realize that there are new APIs or query languages to learn, and new proprietary tools to adopt.New APIs are not just a problem during the learning phase: they also mean vendor lock-in to a proprietary ecosystem. And proprietary ecosystems typically make it hard to export your data.Also, these proprietary SaaS services are often “black boxes”, where your only visibility is the external API, but you have a murky (at best) understanding of the underlying architecture.So the honeymoon period ends quickly, often when you introduce a new type of query, or new workload, or larger data volume, or increased insert or query rate. Or when something even worse happens: the platform starts behaving differently than before, butnothing has changed.Performance debugging is always challenging, but near impossible when you have no lower-level visibility into the underlying system and no mental model of how the underlying systems work.This frustration with black-box services is real. Indeed, when our engineers weretrying to benchmark AWS Timestream (compared to TimescaleDB), this same problem reared its ugly head: AWS Timestream significantly underperformed our expectations (and many others sawsimilar issues), yet our engineers had no idea if AWS Timestream’s performance could be improved, let alone how to do so.Yes, SaaS vendors can blog about their underlying services, but even then, the underlying architecture is probably some complex, polyglot, Rube-Goldberg-like microservice architecture that was never designed for familiarity or understandability.And the general lack of flexibility with serverless data platforms confounds developers as they grow to more complex, production use cases. While many developers might have the same 70-80% needs, it’s the 20-30% that always differ. But enabling this long tail of needs isn’t about just adding more UI buttons to press, it’s about a software architecture that is flexible – that can be customized and optimized for developers’ use cases, yet in a way that a developer can understand. That’s not the case for a SaaS architecture with dozens of subtly interacting microservices.So today’s serverless data platforms are not familiar or flexible. But further, black boxes are never truly easy and worry free: you never know if there are any skeletons lurking in the proverbial closet, just waiting to cause your service to fall over. Or if they will surprise you with unpredictably high costs given all the hidden and opaque charges that often go into monthly consumption charges.Our vision: the database cloudWe’ve alluded to what developers want in their cloud databases: Easy, scalable, familiar, and flexible. Let’s unpack what those mean.Easymeans being able to start with a single click, and then not having to worry about resource sizing, configurations, scale limitations, performance, failures and recovery, upgrades or versioning, security, and more. The service should just work. It should feel easy both to get started and to grow with, both for beginnersandpower users. Serverless data platforms might deliver on “easy for beginners”, but their black-box abstractions fall short on “easy for power users”.Scalablemeans the platform should scale arbitrarily with need. But scalability is not only aboutresources(data volumes, ingest or query rates, or even more subtle issues like data cardinality). It’s also about scalingorganizationally. It should supercharge developer productivity, and grow easily with workflows: from dev/test environments, to production deployments, to sharing data insights across teams, and to multiple applications and use cases within an org. And scalability is finally aboutcost effectiveness– being able to achieve the most performance while keeping your workload within a reasonable budget, including as your workload grows. Today’s serverless platforms scale infrastructurally but not organizationally,nor cost effectively.Familiarmeans not needing to learn a new query language or set of APIs, nor adopt proprietary connectors or tools, nor try (and likely fail) to understand a whole new architecture. Familiarity is how the database scales organizationally, when many developers, product and business owners, and others can already use it. Familiarity also implies that your developers have (or can easily pick up) a clear mental model of the architecture, where they can understand how data is stored, indexed, and processed, they can know when theyshouldbe worrying about the service, and what they can do about it. Serverless platforms are often built around new, custom, proprietary APIs – they are not simply not familiar.Flexiblemeans the database works for more than some limited use case, or more than just under narrow operating conditions, but is a horizontal platform that developers can customize to their needs. It’s not only key-value lookups or basic built-in functions, but supports powerful, rich queries and analytics. It isn’t limited to only storing floats for metrics or in-line labels for tags, but many data types, formats, schemas, indexes. It allows developers to easily trade-off between cost and performance, and to better structure or distribute their data based on need. Flexibility is how the database scales with new use cases: It doesn’t perform well only in some narrow operating conditions, but across a set of applications and workloads. Which also means that expertise gained on one project can be carried forward to the second, third, and tenth projects. Given their lack of such flexibility, serverless platforms fall short here as well.We think of such services as a“transparent box”; easily packaged and accessible to get started, yet with a transparency that provides familiarity and flexibility as you scale. It’s easy and worry-free, not just when starting off butforever, as your workloads and use cases grow.Introducing the new TimescaleToday we are announcing the new Timescale (formerly known as “Timescale Forge”), a database cloud for relational and time-series workloads, built on PostgreSQL and architected around this vision of the “transparent box.”Timescale combines the best of the DBaaS and serverless SaaS data platforms.Unlike DBaaS services, Timescale iseasyandscalable. The platform is built around a modern cloud architecture, with compute and storage fully decoupled. All storage is replicated, encrypted, and highly available: Even if the physical compute hardware fails, the storage stays online and the platform immediately spins up new compute resources, reconnects it to storage, and quickly restores availability.Users can independently resize and scale compute and storage based on their needs (and budget) or set the platform to autoscale storage with their consumption (with autoscaling compute in the works). Whenever a database’s resource configuration changes, the platform automatically re-tunes a user’s database and optimizes it for the new configuration. It’s easy to get started and scale with need.Unlike pure serverless data platforms, Timescale isfamiliarandflexible. It allows developers to build on skills and knowledge they already have with databases. It’s full SQL (not a “SQL-like variant”), the query language they and other teams already know.It works with all the tools, connectors and ORMs, and applications they already use. And it is built on Postgres, so a developer familiar with Postgres (or relational databases more generally) will immediately understand how to use, EXPLAIN, diagnose, and optimize their data models and queries on TimescaleDB. Developers can be immediately productive.In short, a coherent architecture like TimescaleDB, building atop decades of Postgres open-source development and fitting the mental model of developers, enables this in ways that a SaaS platform with an opaque, overly complex software architecture never does.And even though this is the first announcement of our new vision, this product has already been available for the last year (formerly known as “Timescale Forge”), powering production workloads for companies worldwide, including analytics for 10 public transit agencies, music streaming services, smart agriculture, SaaS billing, customer and marketing platforms, building management, supply-chain logistics, real estate, crypto, and many others.""Our experience using Timescale has been fantastic. We're really impressed with the core technical innovations, particularly around hypertables and compression, and it solves a lot of problems for us as we work to build aggregated and derived data sets on top of our core tables. We're excited to see what Timescale continues to build in the future!”- Adam Inoue, Messari (Case Study)""Timescale has been a game-changer for us at Enfinite. Specifically, the Continuous Aggregates and Compression features made querying and storage of large volumes of high-frequency IoT data effortless. The platform’s scalability and ease of use clearly makes it a long-term solution for our database needs.""- Varun Rai, CTO and co-founder ofEnfinite Technologies""Timescale is really helping us scale. Our workload includes queries across both historical and real-time data and our  volume keeps growing and growing, so we were struggling getting enough query speed with vanilla Postgres. Timescale gives us better performance at a much more cost-effective price point than any other solution we looked at.""- Elango Thevar, CEO and co-founder ofNeerKey featuresTimescale is the database cloud for time series and provides all the goodness of TimescaleDB, but now as part of a “transparent box,” that’s just one click away. It includes:Decoupled compute and storagefor maximum flexibility and cost-effectivenessCompute from 0.25 vCPU to 32 vCPU for a wide variety of workloadsStorage volumes from 10GB to 16TB per compute node, all with built-in replicationEffective storage of 100TB+ per node, or petabyte-scale for multi-node deployments, viabest-in-breed compressionfor 94-97% space savingsHigh availability via instant recovery at no additional costOne-click database pause and resumeAutoscaling storagewith zero downtime and configurable limitsAutomated database configuration, yet withfine-grained power-user controlAutomated database re-tuning, whenever resource configurations changeAutomateddata retention policiesfor easy data lifecycle managementAutomatedcontinuous aggregatesto power dashboards and monitoring applicationsAutomateduser-defined actionsfor in-database job schedulingPoint-in-time recovery viaautomated, continuous incremental backupsAutomated, zero-downtime upgrades duringmaintenance windowsExplorer dashboardfor an easy, visual in-console interfaceData encrypted at rest and in transitFlexible role-based database access controlFlexible VPC peering(one-click service migrations from public, dev, test, prod VPCs)Platform observability via metrics and logsDatabase observability into internal statistics, jobs, locking, and moreQuery observability via plan- and execution-time EXPLAINsClose to40 popular PostgreSQL extensionsWorks with any PostgreSQL ORM, connector, or toolTop-rated,highly-technical support team, available 24/7Fully transparent pricing, with fine-grain pricing shown alongside all resource (re)configurationsPlans starting at $39/month with usage-based pricingNew featuresDuring Launch Month this October, Timescale will be getting some great new additions:First-class support formulti-node TimescaleDBservices, with one-click, fully-managed service creation and configurationNew AWS regions across North America and EuropeOne-click database forking, to easily spin up copies of your database for development, testing, and non-prod access for data science teamsAutomated out-of-memory query protection, to eliminate service instability from run-away complex queriesAdvanced billing features, including historical invoicing and configurable billing emails for finance departmentsUpcoming featuresBeyond these new capabilities for this month, the Timescale team is also hard at work on additional capabilities to ship later this quarter, including:Programmatic APIs for flexible control over cloud services, and easily integrate Timescale services into CI/CD pipelines and “infrastructure as code” settingsMulti-node elasticity to support scaling multi-node services up and down with automated data rebalancingHigh availability via service replicas, with automated failure detection and zero downtime failoverRead-only end-points for service replicas to scale read queries while ensuring performance isolation for high-ingest workloadsMulti-user projects for easier collaboration across teamsAnd just wait to see what we have planned for 2022, continuing in our vision of the transparent box for the database cloud. Even easier, more scalable, more familiar, and more flexible.ConclusionTo all the Timescale community members running the 3+ million active TimescaleDB databases today, we thank you again for your support and feedback. We realize how important your data and applications are, and take that trust seriously.To everyone who is not yet a user, and is looking for a database cloud for your relational and time-series workloads, we invite you totry Timescale for free today.If you'd like to connect with Timescale community members, get expert tips, advice and more, tune in toTimescale Community Daylater this month for talks (and demos!) about time-series data and TimescaleDB tips.And, for those who are passionate about data, databases, and delighting developers, and interested in joining a fully-remote, global team:learn about our open positions here. We are hiring broadly across many roles.To the stars! In a transparent, boxy rocket ship! 🐯 🚀Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/announcing-the-new-timescale-cloud-and-a-new-vision-for-the-future-of-database-services-in-the-cloud/
2022-10-06T14:36:05.000Z,"How I Am Planning My Photovoltaic System Using TimescaleDB, Node-RED, and Grafana","Planning photovoltaic systems isn’t easy, even with a specialist at hand. They ask all kinds of questions regarding your power consumption, typical usage hours, or distribution over a year.Collecting consumption data at the granularity of a few seconds is key to finding all the answers for the more precision-loving audience, such as myself.The main reason I spent all this time understanding our actual consumption is simple: cost efficiency. My wife and I have an electricity consumption way out of what is expected of a German household with just two people (and two cats!). That said, I really wanted to get the most out of the system, no matter the cost (well, almost).This is the story of how I used TimescaleDB,Node-RED,Grafana, aRaspberry Pi, some open-source software, and a photodiode to collect data straight from the power meter to plan my photovoltaic system. More than a year after the first data point came in, I have enough insight to answer anything thrown at me—and you can do that too; just carry on reading!Knowing Your Power Consumption and the Lack of SleepAs a normal consumer, at least in Germany, there is almost no way to answer those questions without making stuff up. Most of us send a single meter reading per year for invoicing reasons. That number is divided by 12 months and defines the monthly prepayment. If you paid too much, you’d get some money back. If you paid less than you consumed, you’d get charged the remaining amount.If you want to know the actual value for a per-month consumption, you can write down one monthly number. If you want to know how much you spend in a day, here’s your daily routine. Going further is impractical, though, since hopefully you’ll be sleeping for at least a few hours a day. So, unless you can convince your spouse to take shifts, one number per hour isn’t really achievable.If you ended up here thinking this title is clickbait, allow me to disappoint you. It is all real. At the end of 2019, we bought our house—a nice but older one, built in 1968. From the beginning, we knew we wanted a photovoltaic system on the roof. We’re not the power-saving kind of folks. But while researching, we bumped into a wall of questions to figure out how large (in terms of kWh) the system should be and, even more important, what kind of solar battery storage capacity we should consider.Requirements Are KeySince you have to start somewhere, we set a few basic requirements. The roof needed to be replaced in the next few years so an in-roof system would be interesting. Just for the record, with an in-roof system, the shingle roof is completely replaced with solar panels (a bit like with the Tesla solar tiles). Those things tend to be a bit more expensive but look super cool with the photovoltaic system being the actual roof. That said, we opted for a more expensive system with no shingles in sight. Probably a win-win. :-)Secondly, we knew we wanted battery storage and that it should function as a power outage emergency backup. So, either the batteries themselves or the inverter needed to be able to create their own island power network in case of an outage.Last but not least, it had to support home automation to better integrate with the new heating solution (a hybrid system with gas and heat pump) rather than just switching a relay or two. I also want to make sure that during a power outage certain devices will not be able to power on to prevent an emergency system overload, which has a limited kW budget.What I’m not interested in is providing power back to the net. If the system produces more than I can use or store, fair enough, but I want to delay that as long as possible.Duct Tape, the Professional's Favorite ToolNew power meters (or “Smart Meters” as we call them in Germany;and no, they’re not smart, no joke here!) have built-in digital communication. Using an infrared LED, the system morses out its data in certain intervals. You just™ have to capture the impulses, decode the data, store them, and be done. Thank you for reading.Jokes aside, that is the basic concept. But there are many more non-smart things about the Smart Meter, including a built-in photodiode that reacts to a flashlight blinking (see the link above). Yeah, imagine blinking your four-number pin code. I’m not kidding you!Anyway, the target is clear. We know the steps; let’s get going.The first step is to capture the infrared light impulsed out from the power meter. I built a very simple but super professional setup with a breadboard and duct tape, the professional’s best friend.Using duct tape, I taped the photodiode straight to the power meter’s infrared LED. The breadboard just holds a small resistor. Apart from that, everything’s directly connected to the Raspberry Pi’s UART port.The Raspberry Pi runs the normal Raspbian Light operating system, as well as the decoder software that decodes the SML (Smart Message Language) protocol and forwards it to anMQTT server(for simplicity, I useEclipse Mosquitto, which is already running for my home automation system). The decoder software is called sml2mqtt and is available onGitHub. Big thanks to its developer, spacemanspiff2.A Node-RED workflow handles data transformation and writing into TimescaleDB.MQTT, Node-RED, and TimescaleDBWe now receive many messages in Node-RED at the other end of the MQTT topic. We know each value based on the last segment of the topic’s name. All messages look similar to the following:{
 ""topic"": ""sml2mqtt/090149534b000403de98/watts_l1"",
 ""payload"": ""148"",
 ""qos"": 0,
 ""retain"": false,
 ""_msgid"": ""6a2cd6bb4459e28c""
}As mentioned before, the topic’s name also defines what the payload means. The payload itself is the value, and the other properties are just MQTT or Node-RED elements. We can just ignore them.For our storage, we go with a narrow table setup where one column is used for the values (all of the ones we care for are integers, anyway). We have one column to store the information on what type of value is represented (phase 1-3, total or absolute counter value). Some values are sent more often than others, but we’ll handle that with TimescaleDB’stime_bucketfunction later.To create the necessary metrics table and transform it into a hypertable, we connect to the database (for example, using psql) and execute the following queries:create table metrics
(
   created timestamp with time zone default now() not null,
   type_id integer                                not null,
   value   double precision                       not null
);

select create_hypertable('metrics', 'created');As I want to store data for quite some time, I’ll also go with TimescaleDB’s columnar compression. We are now ready to insert our data through Node-RED.alter table metrics set (timescaledb.compress);
select add_compression_policy('metrics', interval '7 days');That said, the next step is to jump into Node-RED and create a flow. Nothing too complicated, though. The Node-RED flow is (almost) “as simple as it gets.”It takes messages from the MQTT topic, passes them through a switch (with one output per interesting value), does some basic transformation (such as ensuring that the value is a valid integer), moves the type and value into SQL parameters, and eventually calls the actual database insert query with those parameters.As I said, the switch just channels the messages to different outputs depending on the topic’s name. Since I know the last segment of the topic won’t overlap with other topics, I opt for a simple “contains” selector and the name I’m looking for.The functions behind that simply create a JSON object like this:{
 ""type"": $id,
 ""value"": parseInt(msg.payload)
}The term$idis a placeholder for the number of the output (e.g.,watts_l1means1).The second transformer step takes the JSON object and transforms it into an array to be passed directly to the database driver.return {
   params: [msg.type, msg.value]
};Finally, the last node executes the actual database query against the database. Nothing fancy going on here. TimescaleDB uses standard PostgreSQL syntax to write to a hypertable, which means that the full query is just an insert statement, such as:INSERT INTO metrics (type_id, value)
   VALUES ($1, $2);After deploying the flow, it is time to wait. For about a year.Downsampling for ComprehensionNow that we have collected all the data, it is time to start analyzing it. You obviously won’t have to wait for a year. I found it very interesting to keep an eye on changes around weekdays, months, and the different seasons, especially when the heating is on.When analyzing data, it is always important to understand the typical scope or time frame you want to look at. While we have sub-minute (sometimes even to the second) granularity, analyzing at such a micro level is not useful.Using TimescaleDB’s continuous aggregates, we can downsample the information into more comprehensible chunks (chunks are data partitions within a table). I decided that one-hour chunks are granular enough to see the changes in consumption over the day. Apart from that, I also wanted to have daily values.Eventually, I came up with two continuous aggregates to precalculate the necessary data from the actual (real-time) raw values.The first one calculates the kWh per day. Living in Germany, I really want the day line in the correct time zone (Europe/Berlin). That was not yet (easily) possible when I initially built the continuous aggregate,but it is now!create materialized view kwh_day_by_day(time, value)
   with (timescaledb.continuous) as
SELECT time_bucket('1 day', created, 'Europe/Berlin') AS ""time"",
      round((last(value, created) - first(value, created)) * 100.) / 100. AS value
FROM metrics
WHERE type_id = 5
GROUP BY 1;The second continuous aggregate performs the same calculation, but instead of downsampling to a day value, it does it by the hour. While the time zone is not strictly required here, I still find it best to add it for clarity.create materialized view kwh_hour_by_hour(time, value)
   with (timescaledb.continuous) as
SELECT time_bucket('01:00:00', metrics.created, 'Europe/Berlin') AS ""time"",
      round((last(value, created) - first(value, created)) * 100.) / 100. AS value
FROM metrics
WHERE type_id = 5
GROUP BY 1;Finally, it’s time to make things visible.Visual Data AnalyticsFor easier data digestion, I prefer a visual representation. While aggregating data on the command line interface (psql) or with simple query tools would work (the largest resulting dataset would comprise 31 days), it is valuable to have your data presented visually. Especially when you want to use the dashboard with your energy consultant or photovoltaic system engineer.For visualization, I use Grafana. Easy to install, lots of visual plugins, direct support for TimescaleDB (able to generate more specific time-series queries), and I’m just used to it.✨Editor’s Note:If you want to learn how to create awesome visualizations using Grafana and TimescaleDB, check out our Guide to Grafana 101.Over time, the Grafana dashboard has grown with more and more aggregations. Most out of curiosity.Apart from that, the dashboard shows the current consumption per electrical phase, which is updated every few seconds. No live push, but Grafana’s 10-second refresh works great for me.Today, however, I want to focus on the most interesting measurements:Energy consumption by hour of dayEnergy consumption by weekdayEnergy consumption by monthAll those aggregations take the last 12 months into account.Catch Me in My SleepA single day most commonly has 24 hours. Except when you have work, then it probably needs to have 48 hours.Anyway, energy consumption heavily depends on the hour of the day. That said, if you consider buying a battery system, what you want is to provide at least enough capacity to compensate for your night consumption. And remember that the summer days are “longer” (meaning there is sunlight for more hours). Therefore, I consider at least 6 p.m. to 6 a.m. as night hours. Twelve hours to compensate. With the consumption we have, that is quite a bit.But first, let’s figure out how much that is (I’m using the median and the maximum):WITH per_hour AS (
 select
   time,
   value
 from kwh_hour_by_hour
 where ""time"" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year'
 order by 1
), hourly AS (
   SELECT
          extract(HOUR FROM time) * interval '1 hour' as hour,
          value
   FROM per_hour
)
SELECT
   hour,
   approx_percentile(0.50, percentile_agg(value)) as median,
   max(value) as maximum
FROM hourly
GROUP BY 1
ORDER BY 1;Since I’m a big fan of CTE (Common Table Expressions) and find them much more readable than a lot of subqueries, you’ll have to live with it. :-)Anyway, with our already existing continuous aggregation for hour-by-hour consumption, it is as simple as possible to select all of last year’s values. That will result in a list of time buckets and values. The second step is to extract the hour and transform it into an interval (otherwise, Grafana really really won’t like you), and, last but not least, create the actual time-series result set for Grafana to show. Here, I use the Timescale Toolkit functionality for approximate percentiles (which is good enough for me) and tell it to calculate the 50th percentile (the median). The second value is just using the standard PostgreSQL max function.We could stop here because what we just created is our baseline of consumption, which answers the most important question: “What does a common day look like?”For us, as you can see, we wake up just shy of 7 a.m. and take a shower. Since we have hot water through electricity, we can see the consumption increasing quite drastically. You can also see that we head to bed sometime between 11 p.m. to midnight.Finding out the optimal battery capacity is now as simple as adding up either the median or maximum consumption for “night hours,” depending on how much you want to compensate. Remember, it’s also important that the photovoltaic system manages to charge the battery during the day.Weekdays, or the Seven SinsBut we don’t want to stop now. There are two more graphs of interest. Let’s move onward to the aggregation by weekdays.Spoiler alert: there is a difference in consumption. Well, there would be if we were to head to an office to work. But we don’t. Neither me nor my wife. But if you do, there are certainly differences between weekdays and weekends.The query to generate the graph is quite similar to the one before. However, instead of using the hour-by-hour continuous aggregation, we’ll use the day-by-day one. Eventually, we map the values to their respective (human-readable) names and have Grafana render out the time series:WITH per_day AS (
 select
   time,
   value
 from kwh_day_by_day
 where ""time"" at time zone 'Europe/Berlin' > date_trunc('month', time) - interval '1 year'
 order by 1
), daily AS (
   SELECT
       to_char(time, 'Dy') as day,
       value
   FROM per_day
), percentile AS (
   SELECT
       day,
       approx_percentile(0.50, percentile_agg(value)) as value
   FROM daily
   GROUP BY 1
   ORDER BY 1
)
SELECT
   d.day,
   d.ordinal,
   pd.value
FROM unnest(array['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']) WITH ORDINALITY AS d(day, ordinal)
LEFT JOIN percentile pd ON lower(pd.day) = lower(d.day);The result of our work tells us the median consumption per day of the week. One thing I haven’t figured out yet is why Sundays have higher consumption. Maybe the file server is scrubbing the disks; who knows? :-DThe 12 MonthsLast but not least, we also want to see our consumption on a monthly basis. This is what most people collect today by writing down the counter reading on the first day of every month. But since we have the data on a much higher granularity and already have the other graphs, this is as simple as summing up now.I’ll spare you the details, but here’s the query. Nothing to see here. At least nothing we haven’t seen before. :-)WITH per_day AS (
 select
   time,
   value
 from kwh_day_by_day
 where ""time"" > now() - interval '1 year'
 order by 1
), per_month AS (
   SELECT
       to_char(time, 'Mon') as month,
       sum(value) as value
   FROM per_day
   GROUP BY 1
)
SELECT
   m.month,
   m.ordinal,
   pd.value
FROM unnest(array['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']) WITH ORDINALITY AS m(month, ordinal)
LEFT JOIN per_month pd ON lower(pd.month) = lower(m.month)
ORDER BY ordinal;Why Did I Read Until Here?If you made it here, congratulations—and I’m sorry. This got much longer than expected, and I already left out quite a few details. If you have any questions,feel free to reach out. Happy to help and answer questions on the setup.For those amazed at how unprofessional my electrical skills look—I’m scared myself! Jokes aside, this is not the final setup. It was designed for one use case, to collect the information and give me the possibility to make educated decisions. Consider this setup a “temporary workaround.” It will not stay. I promise! Maybe.The interesting fact about all of this is that time series is much more common in our daily lives than some people think. TimescaleDB, in combination with the other tools, made it perfectly easy to set it up, have it running 24/7 and have me quickly make analytics that would have been impossible without a time-series database.Unfortunately, I cannot yet present you with a picture of the ready system. Due to shortages all over the world, the system is still not finished.+Anyway, there are a lot of cool projects and use cases in your home you can use to get started with time-series data, TimescaleDB, and tools such as Grafana and Node-RED.You can sign up for a 30-day free trial here, no credit card required. If you’re interested, check it out! Go!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-i-am-planning-my-photovoltaic-system-using-timescaledb-nodered-and-grafana/
2021-10-01T13:35:34.000Z,"PostgreSQL vs Python for Data Evaluation: What, Why, and How","IntroductionAs I started writing this post, I realized that to properly show how to evaluate, clean, and transform data in the database (also known as data munging), I needed to focus on each step individually. This blog post will show you exactly how to use TimescaleDB and PostgreSQL to perform yourdata evaluation tasksthat you may have previously done in Excel, R, or Python. TimescaleDB and PostgreSQL cannot replace these tools entirely, but they can help your data munging/evaluation tasks be more efficient and, in turn, let Excel, R, and Python shine where they do best: in visualizations, modeling, and machine learning.You may be asking yourself, “What exactly do you mean byEvaluatingthe data?”. When I talk about evaluating the data, I meanreallyunderstanding the data set you are working with.If - in a theoretical world - I could grab a beer with my data set and talk to it about everything, that is what I would do during the evaluating step of my data analysis process. Before beginning analysis, I want to know every column, every general trend, every connection between tables, etc. To do this, I have to sit down and run query after query to get a solid picture of my data.RecapIf you remember,in my last post, I summarized the analysis process as the “data analysis lifecycle” with the following steps: Evaluate, Clean, Transform, and Model.Data Analysis LifecycleAs a data analyst, I found that all the tasks I performed could be grouped into these four categories, with evaluating the data as the first and I feel the most crucial step in the process.My first job out of college was at an energy and sustainability solutions company that focused on monitoring all different kinds of usage - such as electricity, water, sewage, you name it - to figure out how buildings could be more efficient. They would place sensors on whatever medium you wanted to monitor to help you figure out what initiatives your group could take to be more sustainable and ultimately save costs. My role at this company was to perform data analysis and business intelligence tasks.Throughout my time in this job, I got the chance to use many popular tools to evaluate my data, including Excel, R, Python, and heck, even Minitab. But once I tried using a database - and specifically PostgreSQL and TimescaleDB - I realized how efficient and straightforward evaluating work could be when done directly in a database. Lines of code that took me a while to hunt down online, trying to figure out how to accomplish with pandas, could be done intuitively through SQL. Plus, the database queries were just as fast, if not faster, than my other code most of the time.Now, while I would love to show you a one-to-one comparison of my SQL code against each of these popular tools, that’s not practical. Besides, no one wants to read three examples of the same thing in a row! Thus, for comparison purposes in this blog post, I will directly show TimescaleDB and PostgreSQL functionality against Python code. Keep in mind that almost all code will likely be comparable to your Excel and R code. However, if you have any questions, feel free to hop on and join ourSlack channel, where you can ask the Timescale community, or me, specifics on TimescaleDB or PostgreSQL functionality 😊. I’d love to hear from you!Additionally, as we explore TimescaleDB and PostgreSQL functionality together, you may be eager to try things out right away! Which is awesome! If so, you cansign up for a free 30-day trialorinstall and manage TimescaleDB on your own PostgreSQL instances. (You can also learn more byfollowing one of our many tutorials.)But enough of an intro, let’s get into the good stuff!via GIPHYSQL basicsPostgreSQL is a database platform that uses SQL syntax to interact with the data inside it. TimescaleDB is an extension that is applied to a PostgreSQL database. To unlock the potential of PostgreSQL and TimescaleDB, you have to use SQL. So, before we jump into things, I wanted to give a basic SQL syntax refresher.If you are familiar with SQL, please feel free to skip this section!For those of you who are newer to SQL (short for structured query language), it is the language many relational databases, including PostgreSQL, use to query data. Likepandas’ DataFramesor Excel’s spreadsheets, data queried with SQL is structured as a table with columns and rows.Thebasics of a SQLSELECTcommandcan be broken down like this 👇SELECT --columns, functions, aggregates, expressions that describe what you want to be shown in the results
FROM --if selecting data from a table in your DB, you must define the table name here
JOIN --join another table to the FROM statement table 
    ON --a column that each table shares values 
WHERE --statement to filter results where a column or expression is equivalent to some statement
GROUP BY --if SELECT or WHERE statement contains an aggregate, or if you want to group values on a column/expression, must include columns here
HAVING --similar to WHERE, this keyword helps to filter results based upon columns or expressions specifically used with a GROUP BY query
ORDER BY --allows you to specify the order in which your data is displayed
LIMIT --lets you specify the number of rows you want displayed in the outputYou can think of your queries asSELECTingdataFROMyour tables within your database. You canJOINmultiple tables together and specifyWHEREyour data needs to be filtered or what it should beGROUPed BY. Do you see what I did there 😋?This is the beauty of SQL; these keyword's names were chosen to make your queries intuitive. Thankfully, most of PostgreSQL and SQL functionality follow this same easy-to-read pattern. I have had the opportunity to teach myself many programming languages throughout my career, and SQL is by far the easiest to read, write, and construct. This intuitive nature is another excellent reason why data munging in PostgreSQL and TimescaleDB can be so efficient when compared to other methods.Note that this list of keywords includes most of the ones you will need to start selecting data with SQL; however, it is not exhaustive. You will not need to use all these phrases for every query but likely will need at leastSELECTandFROM. The queries in this blog post will always include these two keywords.Additionally, the order of these keywords is specific. When building your queries, you need to follow the order that I used above. For any additional PostgreSQL commands you wish to use, you will have to research where they fit in the order hierarchy and follow that accordingly.Seeing a list of commands may be somewhat helpful but is likely not enough to solidify understanding if you are like me. So let’s look at some examples!Let’s say that I have a table in my PostgreSQL database calledenergy_usage. This table contains three columns:timewhich contains timestamp values,energywhich contains numeric values, andnoteswhich contains string values. As you may be able to imagine, every row of data in myenergytable will contain,time: timestamp value saying when the reading was collectedenergy: numeric value representing how much energy was used since the last readingnotes: string value giving additional context to each reading.If I wanted to look at all the data within the table, I could use the following SQL querySELECT time, energy, notes --I list my columns here
FROM energy_usage ;-- I list my table here and end query with semi-colonAlternatively, SQL has a shorthand for ‘include all columns’, the operator*. So I could select all the data using this query as well,SELECT *
FROM energy_usage;What if I want to select the data and order it by thetimecolumn so that the earliest readings are first and the latest are last? All I need to do is include theORDER BYstatement and then specify thetimecolumn along with the specificationASCto let the database know I want the data in ascending order.SELECT *
FROM energy_usage
ORDER BY time ASC;-- first I list my time column then I specify either DESC or ASCHopefully, you can start to see the pattern and feel more comfortable with SQL syntax. I will be showing a lot more code snippets throughout the post, so hang tight if you still need more examples!So now that we have a little refresher on SQL basics, let’s jump into how you can use this language along with TimescaleDB and PostgreSQL functionality to do your data evaluating tasks!A quick note on the dataEarlier I talked about my first job as a data analyst for an IoT sustainability company. Because of this job, I tend to love IoT data sets and couldn’t pass up the chance to explorethis IoT dataset from Kaggleto show how to perform data munging tasks in PostgreSQL and TimescaleDB.The data set contains two tables, one specifying energy consumption for a single home in Houston, Texas (calledpower_usage), and the other documenting weather conditions (calledweather). This data is actually the same data set that I used in my previous post, so bonus points if you caught that 😊!This data was recorded from January 2016 to December 2020. While looking at this data set, and all time-series data sets, we must consider any outside influences that could affect the data. The most obvious factor that impacts the analysis of this dataset is the COVID-19 pandemic that took place from January 9th through to December 2020. Thankfully, we will see that the individual recording this data included some notes to help categorize days affected by the pandemic. As I go through this blog series, we will see patterns associated with the data collected during the COVID-19 pandemic, so definitely keep this fact in the back of your mind as we perform various data munging analysis steps!Here is an image explaining the two tables, their column names in red and corresponding data types in blue.As we work through this blog post, we will use the evaluating techniques available within PostgreSQL and TimescaleDB to understand these two tables inside and out.Evaluating the dataAs we discussed before, the first step in the data analysis lifecycle - and arguably the most critical step -  is to evaluate the data. I will go through how I would approach evaluating this IoT energy data, showing most of the techniques I have used in the past while working in data science. While these examples are not exhaustive, they will cover many of the evaluating steps you perform during your analysis, helping to make your evaluating tasks more efficient by using PostgreSQL and TimescaleDB.The techniques that I will cover include:Reading the raw dataFinding and observing “categorical” column values in my datasetSorting my data by specific columnsDisplaying grouped dataFinding abnormalities in the databaseLooking at general trendsReading the raw dataLet’s start with the most simple evaluating task, looking at the raw data.As we learned in the SQL refresher above, we can quickly pull all the data within a table by using theSELECTstatement with the*operator. Since I have two tables within my database, I will query both table’s information by running a query for each.PostgreSQL code:-- select all the data from my power_usage table
SELECT * 
FROM power_usage pu; 
-- selects all the data from my weather table
SELECT * 
FROM weather w;But what if I don’t necessarily need to query all my data? Since all the data is housed in the database, if I want to get a feel for the data and the column values, I could just look at a snapshot of the raw data.While conducting analysis in Python, I often would just print a handful of rows of data to get a feel for the values. We can do this in PostgreSQL by including theLIMITcommand within our query. To show the first 20 rows of data in my tables, I can do the following:PostgreSQL code:-- select all the data from my power_usage table
SELECT * 
FROM power_usage pu
LIMIT 20; -- specify 20 because I only want to see 20 rows of data
-- selects all the data from my weather table
SELECT * 
FROM weather w 
LIMIT 20;Results: Some of the rows for each tablestartdatevalue_kwhday_of_weeknotes2016-01-06 01:00:0012weekday2016-01-06 02:00:0012weekday2016-01-06 03:00:0012weekday2016-01-06 04:00:0012weekday2016-01-06 05:00:0002weekday2016-01-06 06:00:0002weekdaydatedaytemp_maxtemp_avgtemp_mindew_maxdew_avgdew_minhum_maxhum_avghum_minwind_maxwind_avgwind_minpress_maxpress_avgpress_minprecipitday_of_week2016-01-061857568747166100896521100303030022016-02-06276716674706610097891880303030452016-02-0729586767673699467431260303030062016-02-0829787777774719466431550303030002016-02-0929585777574709070511670303030012016-02-102867465646158906640840303030022016-03-0637972687270681009472185030303036Python code:In this first Python code snippet, I show the modules I needed to import and the connection code that I would have to run to access the data from my database and import it into a pandas DataFrame.One of the challenges I faced while data munging in Python was the need to run through the entire script again and again when evaluating, cleaning, and transforming the data. This initial data pulling process usually takes a good bit of time, so it was often frustrating to run through it repetitively. I also would have to run print anytime I wanted to quickly glance at an array, Dataframe, or element. These kinds of extra tasks in Python can be time-consuming, especially if you end up at the modeling stage of the analysis lifecycle with only a subset of the original data! All this to say, keep in mind that for the other code snippets within the blog, I will not include this as part of the code; however, it still impacts that code in the background.Additionally, because I have my data housed in a TimescaleDB instance, I still need to use theSELECTstatement to query the data from the database and read it into Python. If you use a relational database - which I explained is very beneficial to analysis in my previous post - you will have to usesomeSQL.import psycopg2
import pandas as pd
import configparser
import numpy as np
import tempfile
import matplotlib.pyplot as plt
 
## use config file for database connection information
config = configparser.ConfigParser()
config.read('env.ini')
 
## establish conntection
conn = psycopg2.connect(database=config.get('USERINFO', 'DB_NAME'),
                       host=config.get('USERINFO', 'HOST'),
                       user=config.get('USERINFO', 'USER'),
                       password=config.get('USERINFO', 'PASS'),
                       port=config.get('USERINFO', 'PORT'))
 

## define the queries for copying data out of our database (using format to copy queries)                    
query_weather = ""select * from weather""
query_power = ""select * from power_usage""
## define function to copy the data to a csv
def copy_from_db(query, cur):
    with tempfile.TemporaryFile() as tmpfile:
        copy_sql = ""COPY ({query}) TO STDOUT WITH CSV {head}"".format(
            query=query, head=""HEADER""
            )
        cur.copy_expert(copy_sql, tmpfile)
        tmpfile.seek(0)
        df = pd.read_csv(tmpfile)
        return df
## create cursor to use in function above and place data into a file
cursor = conn.cursor()
weather_df = copy_from_db(query_weather, cursor)
power_df = copy_from_db(query_power, cursor)
cursor.close()
conn.close()


print(weather_df.head(20))
print(power_df.head(20))Finding and observing “categorical” column values in my datasetNext, I think it is essential to understand any “categorical” columns - columns with a finite set of values - that I might have. This is useful in analysis because categorical data can give insight into natural groupings that often occur within a dataset. For example, I would assume that energy usage for many people is different on a weekday vs. a weekend. We can’t verify this without knowing the categorical possibilities and seeing how each could impact the data trend.First, I want to look at my tables and the data types used for each column. Looking at the available columns in each table, I can make an educated guess that theday_of_week,notes, anddaycolumns will be categorical. Let’s find out if they indeed are and how many different values exist in each.To find all the distinct values within a column (or between multiple columns), you can use theDISTINCTkeyword afterSELECTin your query statement. This can be useful for several data munging tasks, such as identifying categories - which I need to do - or finding unique sets of data.Since I want to look at the unique values within each column individually, I will run a query for each separately. If I were to run a query like this 👇SELECT DISTINCT day_of_week, notes 
FROM power_usage pu;I would get data like thisday_of_weeknotes3vacation3weekday1weekday1vacation2vacation4vacationThe output data would show uniquepairsofday_of_weekandnotesrelatedvalues within the table. This is why I need to include a single column in each statement so that I only see that individual column’s unique values and not the unique sets of values.For these queries, I am also going to include theORDER BYcommand to show the values of each column in ascending order.PostgreSQL code:-- selecting distinct values in the ‘day_of_week’ column within my power_usage table
SELECT DISTINCT day_of_week 
FROM power_usage pu 
ORDER BY day_of_week ASC;
-- selecting distinct values in the ‘notes’ column within my power_usage table
SELECT DISTINCT notes 
FROM power_usage pu 
ORDER BY notes ASC;

-- selecting distinct values in the ‘day’ column within my weather table
SELECT DISTINCT ""day"" 
FROM weather w 
ORDER BY ""day"" ASC;
-- selecting distinct values in the ‘day_of_week’ column within my weather table
SELECT DISTINCT day_of_week 
FROM weather w 
ORDER BY day_of_week ASC;Results:Notice that we see the recorder for this data included “COVID-19” as a category in theirnotescolumn. As mentioned above, this note could be necessary to finding and understanding patterns in this family's energy usage.day_of_week0123456notesCOVID_lockdownvacationweekdayweekend(Only some of the values shown for day)day12345678910Python code:In my Python code, notice that I need to print anything that I want to quickly observe. I have found this to be the quickest solution, even when compared to using the Python console in debug mode.p_day_of_the_week = power_df['day_of_week'].unique()
p_notes = power_df['notes'].unique()
w_day = weather_df['day'].unique()
w_day_of_the_week = power_df['day_of_week'].unique()
print(sorted(p_day_of_the_week), sorted(p_notes), sorted(w_day), sorted(w_day_of_the_week))Sorting my data by specific columnsWhat if I want to evaluate my tables based on how specific columns were sorted? One of the top questions asked on StackOverflow for Python data analysis is ""How to sort a dataframe in python pandas by two or more columns?"". Once again, we can do this intuitively through SQL.One of the things I'm interested in identifying is how bad weather impacts energy usage. To do this, I have to think about indicators that typically signal bad weather, which include high precipitation, high wind speed, and low pressure. To identify days with this pattern in my PostgreSQLweathertable, I need to use theORDER BYkeyword, then call out each column in the order I want things sorted, specifying theDESCandASCattributes as needed.PostgreSQL code:-- sort weather data by precipitation desc first, wind_avg desc second, and pressure asc third
SELECT ""date"", precipit, wind_avg, press_avg 
FROM weather w 
ORDER BY precipit DESC, wind_avg DESC, press_avg ASC;Results:dateprecipitwind_avgpress_avg2017-08-271315302017-08-281124302019-09-2099302017-08-0865302017-08-29522302018-08-12512302016-02-0648302018-05-0747302019-10-0539302018-03-2938302016-03-0635302018-06-19212302019-08-05211302019-10-3021130Python code:I have often found the different pandas or Python functions to be harder to know off the top of my head. With how popular the StackOverflow question is, I can imagine that many of you also had to refer to Google for how to do this initially.sorted_weather = weather_df[['date', 'precipit', 'wind_avg', 'press_avg']].sort_values(['precipit', 'wind_avg', 'press_avg'], ascending=[False, True, False])
print(sorted_weather)Displaying grouped dataFinding the sum of energy usage from data that records energy per hour can be instrumental in understanding data patterns. This concept boils down to performing a type of aggregation over a particular column. Between PostgreSQL and TimescaleDB, we have access to almost every type of aggregation function we could need. I will show some of these operators in this blog series, but I strongly encourage all of you tolookup morefor your own use!From the categorical section earlier, I mentioned that I suspect people could have different energy behavior patterns on weekdays vs. weekends, particularly in a single-family home in the US. Given my data set, I’m curious about this hypothesis and want to find the cumulative energy consumption across each day of the week.To do so, I need to sum all the kWh data (value_kwh) in the power table, then group this data by the day of the week (day_of_week). In order to sum my data in PostgreSQL, I will use theSUM()function. Because this is an aggregation function, I will have to include something that tells the database what to sum over. Since I want to know the sum of energy over each type of day, I can specify that the sum should be grouped by theday_of_weekcolumn using theGROUP BYkeyword. I also added theORDER BYkeyword so that we could look at the weekly summed usage in order of the day.PostgreSQL code:-- first I select the day_of_week col, then I define SUM(value_kwn) to get the sum of value_kwh col
SELECT day_of_week, SUM(value_kwh) --sum the value_sum column
FROM power_usage pu 
GROUP BY day_of_week -- group by the day_of_week col
ORDER BY day_of_week ASC; -- decided to order data by the day_of_week ascResults:After some quick investigation, the value0in theday_of_weekcolumn represents a Monday, thus my hypothesis may just be right.day_of_weeksum03849139592394734094439875416964311Python code:Something to note about the pandasgroupby()function is that the group by column in the DataFrame will become the index column in the resulting aggregated DataFrame. This can add some extra work later on.day_agg_power = power_df.groupby('day_of_week').agg({'value_kwh' : 'sum'})
print(day_agg_power)Finding abnormalities in the databaseClean data is fundamental in producing accurate analysis, and abnormalities/errors can be a huge roadblock to clean data. An essential part of evaluating data is finding abnormalities to determine if an error caused them. No data set is perfect, so it is vital to hunt down any possible errors in preparation for the cleaning stage of our analysis. Let's look at one example of how to uncover issues in a dataset using our example energy data.After looking at the raw data in mypower_usagetable, I found that thenotesandday_of_weekcolumnsshould be the same for each hour across a single day(there are 24 hourly readings each day, and each hour is supposed to have the samenotesvalue). In my experience with data analysis, I have found that notes which need to be recorded granularly often have mistakes within them. Because of this, I wanted to investigate whether or not this pattern was consistent across all of the data.To check this hypothesis I can use the TimescaleDBtime_bucket()function, PostgreSQL’sGROUP BYkeyword, andCTEs(common table expressions). While theGROUP BYkeyword is likely familiar to you by now, CTEs and thetime_bucket()function are not. So, before I show the query, let’s dive into these two features.Time bucket functionThetime_bucket()function allows you to take a timestamp column likestartdatein thepower_usagetable, and “bucket” the time based on the interval of your choice. For example,startdateis a timestamp column that shows values for each hour in a day. You could use thetime_bucket()function on this column to “bucket” the hourly data into daily data.Here is an image that shows how rows of thestartdatecolumn are bucketed into one aggregate row withtime_bucket(‘1 day’, startdate).After using thetime_bucket()function in my query, I will have one unique “date” value for any data recorded over a single day. Sincenotesandday_of_weekshould also be unique over each day, if Igroup bythese columns, I should get a single set of (date, day_of_week, notes) values.Notice that to useGROUP BYin this scenario, I just list the columns I want to group on. Also, notice that I addedASafter mytime_bucket()function, this keyword allows you to ""rename"" columns. In the results, look for thedaycolumn, as this comes directly from my rename.PostgreSQL code:-- select the date through time_bucket and get unique values for each 
-- (date, day_of_week, notes) set
SELECT 
	time_bucket(interval '1 day', startdate ) AS day,
	day_of_week,
	notes
FROM power_usage pu 
GROUP BY day, day_of_week, notes;Results: Some of the rowsdayday_of_weeknotes2017-01-19 00:00:003weekday2016-10-06 00:00:003weekday2017-06-04 00:00:006weekend2019-01-03 00:00:003weekday2017-10-01 00:00:006weekend2019-11-27 00:00:002weekday2017-06-15 00:00:003weekday2016-11-16 00:00:002weekday2017-05-18 00:00:003weekday2018-07-17 00:00:001weekday2020-03-06 00:00:004weekday2018-10-14 00:00:006weekendPython code:In my Python code, I cannot just manipulate the table to print results, I actually have to create another column in the DataFrame.day_col = pd.to_datetime(power_df['startdate']).dt.strftime('%Y-%m-%d')
power_df.insert(0, 'date_day', day_col)
power_unique = power_df[['date_day', 'day_of_week', 'notes']].drop_duplicates()
print(power_unique)Now that we understand thetime_bucket()function a little better, let's look at CTEs and how they help me use this bucketed data to find any errors within thenotescolumn.CTEs or common table expressionsGetting unique sets of data only solves half of my problem. Now I want to verify if each day is truly mapped to a singleday_of_weekandnotespair. This is where CTE’s come in handy. With CTEs, you can build a query based on the results of others.CTE’s use the following format 👇WITH query_1 AS (
SELECT -- columns expressions
FROM table_name
)
SELECT --column expressions 
FROM query_1;WITHandASallow you to define the first query, then in the secondSELECTstatement, you can call the results from the first query as if it were another table in the database.To check that each day was “mapped” to a singleday_of_weekandnotespair, I need to aggregate the queriedtime_bucket()table above based upon the date column using another PostgreSQL aggregation functionCOUNT(). I am doing this because each dayshouldonly count one uniqueday_of_weekandnotespair. If the count results in two or more, this implies that one day contains multipleday_of_weekandnotespairs and thus is showing abnormal data.Additionally, I will add aHAVINGstatement into my query so that the output only displays rows where theCOUNT(day)is greater than one. I will also throw in anORDER BYstatement in case we have many different values greater than 1.PostgreSQL code:WITH power_unique AS (
-- query from above, get unique set of (date, day_of_week, notes)
SELECT 
	time_bucket(INTERVAL '1 day', startdate ) AS day,
	day_of_week,
	notes
FROM power_usage pu 
GROUP BY day, day_of_week, notes
)
-- calls data from the query above, using the COUNT() agg function
SELECT day, COUNT(day) 
FROM power_unique
GROUP BY day
HAVING COUNT(day) > 1
ORDER BY COUNT(day) DESC;Results:daycount2017-12-27 00:00:0022020-01-03 00:00:0022018-06-02 00:00:0022019-06-03 00:00:0022020-07-01 00:00:0022016-07-21 00:00:002Python code:Because of the count aggregation, I needed to rename the column in myagg_power_uniqueDataFrame so that I could then sort the values.day_col = pd.to_datetime(power_df['startdate']).dt.strftime('%Y-%m-%d')
## If you ran the previous code snippet, this next line will error since you already ran it
power_df.insert(0, 'date_day', day_col)
power_unique = power_df[['date_day', 'day_of_week', 'notes']].drop_duplicates()
agg_power_unique = power_unique.groupby('date_day').agg({'date_day' : 'count'})
agg_power_unique = agg_power_unique.rename(columns={'date_day': 'count'})
print(agg_power_unique.loc[agg_power_unique['count'] > 1].sort_values('count', ascending=False))This query reveals that I indeed have a couple of data points that seem suspicious. Specifically, the dates [2017-12-27, 2020-01-03, 2018-06-02, 2019-06-03, 2020-07-01, 2016-07-21]. I will demonstrate how to fix these date issues in a later blog post about Cleaning techniques.This example only shows one set of functions which helped me identify abnormal data through grouping and aggregation. You can use many other PostgreSQL and TimescaleDB functions to find other abnormalities in your data, like utilizing TimescaleDB’sapprox_percentile()function (introducing this next) to find outliers in numeric columns by playing around with interquartile range calculations.Looking at general trendsArguably, one of the more critical aspects of evaluating your data is understanding the general trends. To do this, you need to get basic statistics on your data using functions like mean, interquartile range, maximum values, and others. TimescaleDB has created many optimized hyperfunctions to perform these very tasks.To calculate these values, I am going to introduce the following TimescaleDB functions: `approx_percentile`, `min_val`, `max_val`, `mean`, `num_vals`,`percentile_agg` (aggregate), and `tdigest` (aggregate)These hyperfunctions fall under the TimescaleDB category of two-step aggregation. Timescale designed each function to either be an aggregate or accessor function (I noted which ones above were aggregate functions). In two-step aggregation, the more programmatically taxing aggregate function is calculated first, then the accessor function is applied to it after.For specifics on how two-step aggregation works and why we use this convention, check outDavid Kohn’s blog series on our hyperfunctions and two-step aggregation.I definitely want to understand the basic trends within thepower_usagetable for my data set. If I plan to do any type of modeling to predict future usage trends, I need to know some basic information about what this home’s usage looks like daily.To understand the daily power usage data distribution, I’ll need to aggregate the energy usage per day. To do this, I can use thetime_bucket()function I mentioned above, along with theSUM()operator.-- bucket the daily data using time_bucket, sum kWh over each bucketed day
SELECT 
	time_bucket(INTERVAL '1 day', startdate ) AS day,
	SUM(value_kwh)
FROM power_usage pu 
GROUP BY day;I then want to find the 1st, 10th, 25th, 75th, 90th, and 99th percentiles, the median or 50th percentile, mean, minimum value, maximum value, number of readings in the table, and interquartile range of this data. Creating the query with a CTE simplifies the process by only calculating the sum of data once and reusing the value multiple times.PostgreSQL:WITH power_usage_sum AS (
-- bucket the daily data using time_bucket, sum kWh over each bucketed day
SELECT 
	time_bucket(INTERVAL '1 day', startdate ) AS day,
	SUM(value_kwh) AS sum_kwh
FROM power_usage pu 
GROUP BY day
)
-- using two-step aggregation functions to find stats
SELECT approx_percentile(0.01,percentile_agg(sum_kwh)) AS ""1p"",
approx_percentile(0.10,percentile_agg(sum_kwh)) AS ""10p"",
approx_percentile(0.25,percentile_agg(sum_kwh)) AS ""25p"",
approx_percentile(0.5,percentile_agg(sum_kwh)) AS ""50p"",
approx_percentile(0.75,percentile_agg(sum_kwh)) AS ""75p"",
approx_percentile(0.90,percentile_agg(sum_kwh)) AS ""90p"",
approx_percentile(0.99,percentile_agg(sum_kwh)) AS ""99p"",
min_val(tdigest(100, sum_kwh)),
max_val(tdigest(100, sum_kwh)),
mean(percentile_agg(sum_kwh)),
num_vals(percentile_agg(sum_kwh)),
-- you can use subtraction to create an output for the IQR
approx_percentile(0.75,percentile_agg(sum_kwh)) - approx_percentile(0.25,percentile_agg(sum_kwh)) AS iqr
FROM power_usage_sum pus;Results:1p10p25p50p75p90p99pmin_valmax_valmeannum_valsiqr0.04.00286.993616.006628.991438.978156.99710.073.018.90251498.021.9978Python:Something that really stumped me when initially writing this code snippet was that I had to useastype(float)on myvalue_kwhcolumn to use describe. I have probably spent the combined time of a day over my life trying to deal with value types being incompatible with certain functions. This is another reason why I enjoy data munging with the intuitive functionality of PostgreSQL and TimescaleDB; these types of problems just happen less often. And let me tell you, the faster and painless data munging is the happier I am!agg_power = power_df.groupby('date_day').agg({'value_kwh' : 'sum'})
# need to make the value_kwh column the right data type
agg_power.value_kwh = agg_power.value_kwh.astype(float)
describe = agg_power.value_kwh.describe()
percentiles = agg_power.value_kwh.quantile([.01, .1, .9, .99])
q75, q25 = np.percentile(agg_power['value_kwh'], [75 ,25])
iqr = q75 - q25
print(describe, percentiles, iqr)Another technique you may want to use for accessing the distribution of data in a column is a histogram. Generally, creating an image is where Python and other tools shine. However, I often need to glance at a histogram to check for any blatant anomalies when evaluating data. While this one technique in TimescaleDB may not be as simple as the Python solution, I can still do this directly in my database, which can be convenient.To create a histogram in the database, we will need to use the TimescaleDBhistogram()function,unnest(),generate_series(),repeat(), and CTE’s.Thehistogram()function takes in the column you want to analyze and produces an array object which contains the frequency values across the number of buckets plus two (one additional bucket for values below the lowest bucket and above the highest bucket). You can then use PostgreSQL’sunnest()function to break up the array into a single column with rows equal to two plus the number of specified buckets.Once you have a column with bucket frequencies, you can then create a histogram “image” using the PostgreSQLrepeat()function. The first time I saw someone use therepeat()function in this way was inHaki Benita’s blog post, which I recommend reading if you are interested in learning more PostgreSQL analytical techniques. Therepeat()function essentially creates a string that repeats chosen characters a specified number of times. To use the histogram frequency values, you just input the unnested histogram in for the repeating argument.Additionally, I find it useful to know the approximate starting values for each bucket in the histogram. This gives me a better picture of what values are occurring when. To approximate the bin values, I use the PostgreSQLgenerate_series()function along with some algebra,(generate_series(-1, [number_of_buckets]) * [max_val - min_val]::float/[number_of_buckets]::float) + [min_val]When I put all these techniques together, I am able to get a histogram with the following,PostgreSQL:WITH power_usage_sum AS (
-- bucket the daily data using time_bucket, sum kWh over each bucketed day
SELECT 
	time_bucket(INTERVAL '1 day', startdate ) AS day,
	SUM(value_kwh) AS sum_kwh
FROM power_usage pu 
GROUP BY day
),
histogram AS (
-- I input the column = sum_kwh, the min value = 0, max value = 73, and number of buckets = 30
SELECT histogram(sum_kwh, 0, 73, 30)
FROM power_usage_sum w 
)
SELECT 
-- I use unnest to create the first column
   unnest(histogram) AS count, 
-- I use my approximate bucket values function
   (generate_series(-1, 30) * 73::float/30::float) + 0 AS approx_bucket_start_val,
-- I then use the repeat function to display the frequency
   repeat('■', unnest(histogram)) AS frequency
FROM histogram;Results:countapprox_bucket_start_valfrequency0-2.43830.0■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■1042.43■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■2074.87■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■1057.3■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■1509.73■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■7612.17■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■10514.6■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■6217.03■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■4819.47■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■7721.9■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■3524.33■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■8326.77■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■4229.2■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■7231.63■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■4634.07■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■5136.5■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■3938.93■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■3241.37■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■2443.8■■■■■■■■■■■■■■■■■■■■■■■■1646.23■■■■■■■■■■■■■■■■1748.67■■■■■■■■■■■■■■■■■451.1■■■■353.53■■■555.97■■■■■458.4■■■■560.83■■■■■163.27■065.7168.13■070.57173.0■Python:This Python code is definitively better. It’s simple and relatively painless. I wanted to show this comparison to provide an option for displaying a histogram directly in your database vs. having to pull the data into a pandas DataFrame then displaying it. Doing the histogram in the database just helps me to keep focus while evaluating the data.plt.hist(agg_power.value_kwh, bins=30)
plt.show()Wrap UpHopefully, after reading through these various evaluating techniques, you feel more comfortable with exploring some of the possibilities that PostgreSQL and TimescaleDB provide. Evaluating data directly in the database often saved me time without sacrificing any functionality. If you are looking to save time and effort while evaluating your data for analysis, definitely consider using PostgreSQL and TimescaleDB.In my next posts, I will go over techniques to clean and transform data using PostgreSQL and TimescaleDB. I'll then take everything we've learned together to benchmark data munging tasks in PostgreSQL and TimescaleDB vs. Python and pandas. The final blog post will walk you through the full process on a real dataset by conducting deep-dive data analysis with TimescaleDB (for data munging) and Python (for modeling and visualizations).If you have questions about TimescaleDB, time-series data, or any of the functionality mentioned above, join ourcommunity Slack, where you'll find an active community of time-series enthusiasts and various Timescale team members (including me!).If you’re ready to see the power of TimescaleDB and PostgreSQL right away, you cansign up for a free 30-day trialorinstall TimescaleDB and manage it on your current PostgreSQL instances. We also have a bunch ofgreat tutorialsto help get you started.Until next time!Functionality GlossarySELECTFROMORDER BYDESCASCLIMITDISTINCTGROUP BYSUM()time_bucket(<time_interval>, <time_col>)CTE’sWITHASCOUNT()approx_percentile()min_val()max_val()mean()num_vals()percentile_agg()[aggregate]tdigest()[aggregate]histogram()unnest()generate_series()repeat()Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-evaluate-your-data-directly-within-the-database-and-make-your-analysis-more-efficient/
2023-03-08T14:00:10.000Z,The Power of Linked Data Event Streams and Timescale for Real-Time Management of Time-Series Data,"This guest blog post was originally publishedin the author’s blog. Thank you so much to Samuel for sending it to us for publication in the Timescale Blog.What Is a Linked Data Event Stream?Linked Data Event Streams (LDES) represent and share fast and slow-moving data on the Web using theResource Description Framework(RDF), which allows data to be linked and connected to other data sources using unique identifiers (URIs).A Linked Data Event Stream is a data event stream of a group of immutable objects described as machine-readable RDF (such as sensor observations, address registers, or financial data).LDES streams provide a flexible and interoperable way of describing and exchanging events as Linked Data, enabling different systems and applications to easily consume and act on data streams in a consistent and standardized way.This article shows how to effortlessly insert sensor data in the form of an LDES into a Timescale database.Managing Large Amounts of Time-Series DataTimescaleis an open-source database for storing and querying large amounts of time-series data. It extends PostgreSQL with time-series support, offering features like fast ingestion and querying of large data, flexible data granularity, and long-term data storage.Insertperformance comparison between TimescaleDB 2.7.2 and PostgreSQL 14.4Timescale is a powerful and efficient database system well-suited for storing and querying time-series data at scale. It is widely used in various applications, including IoT, finance, and telemetry.First, you need to configure a data flow to ingest a Linked Data Event Stream into PostgreSQL. You can do this in anApache NiFienvironment.LDES to TimescaleTo persist LDES in a database, we use the Apache NiFi platform. Apache NiFi is an open-source data processing and integration platform designed to automate data flows between systems. It provides a Web-based user interface for creating, managing, and monitoring data flows and a range of pre-built connectors and processors for performing data processing tasks.Data pipeline in Apache NiFiTo consume an LDES stream, an LDES client processor is needed in the Apache NiFi flow. An LDES client is a component capable of consuming, processing, and analyzing the events in the stream.The output of the LDES Client (working with the followingLDES):_:B5edf92f59913b8d14f45818d5bde1d51 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement> .
_:B5edf92f59913b8d14f45818d5bde1d51 <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.observedProperty> <https://data.vmm.be/concept/waterkwaliteitparameter/conductiviteit> .
_:B5edf92f59913b8d14f45818d5bde1d51 <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.phenomenonTime> ""2022-11-09T20:30:00.000Z""^^<http://www.w3.org/2001/XMLSchema#datetime> .
_:B5edf92f59913b8d14f45818d5bde1d51 <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.result> _:B727a9b2b6a6311cc83b677d439f4cf68 .
_:B5edf92f59913b8d14f45818d5bde1d51 <http://www.w3.org/ns/sosa/madeBySensor> <urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j> .
_:B6003104aae33209ee0e6a26e14ba38cb <https://schema.org/value> ""1.152E1""^^<http://www.w3.org/2001/XMLSchema#double> .
_:Bec20a7e293639ee5a985d621d7b2d6d5 <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement> .
_:Bec20a7e293639ee5a985d621d7b2d6d5 <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.observedProperty> <https://data.vmm.be/concept/waterkwaliteitparameter/temperatuur> .
_:Bec20a7e293639ee5a985d621d7b2d6d5 <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.phenomenonTime> ""2022-11-09T20:30:00.000Z""^^<http://www.w3.org/2001/XMLSchema#datetime> .
_:Bec20a7e293639ee5a985d621d7b2d6d5 <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.result> _:Bdd20913615f9b4c0169c7770bdc770d4 .
_:Bec20a7e293639ee5a985d621d7b2d6d5 <http://www.w3.org/ns/sosa/madeBySensor> <urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j> .
_:Bdd20913615f9b4c0169c7770bdc770d4 <http://def.isotc211.org/iso19103/2005/UnitsOfMeasure#Measure.value> _:B6003104aae33209ee0e6a26e14ba38cb .
<urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://www.w3.org/TR/vocab-ssn-ext/#sosa:ObservationCollection> .
<urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z> <http://def.isotc211.org/iso19156/2011/SamplingFeature#SF_SamplingFeatureCollection.member> _:B5edf92f59913b8d14f45818d5bde1d51 .
<urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z> <http://def.isotc211.org/iso19156/2011/SamplingFeature#SF_SamplingFeatureCollection.member> _:B3105a27f0059867877fb28653f5a6abc .
<urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z> <http://def.isotc211.org/iso19156/2011/SamplingFeature#SF_SamplingFeatureCollection.member> _:Bec20a7e293639ee5a985d621d7b2d6d5 .
<urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z> <http://purl.org/dc/terms/isVersionOf> <urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn> .
<urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z> <http://www.w3.org/ns/prov#generatedAtTime> ""2022-11-09T20:30:00.000Z""^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z> <http://www.w3.org/ns/sosa/hasFeatureOfInterest> ""spt-00029-97"" .
_:B3105a27f0059867877fb28653f5a6abc <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://def.isotc211.org/iso19156/2011/Measurement#OM_Measurement> .
_:B3105a27f0059867877fb28653f5a6abc <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.observedProperty> <https://data.vmm.be/concept/observatieparameter/hydrostatische-druk> .
_:B3105a27f0059867877fb28653f5a6abc <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.phenomenonTime> ""2022-11-09T20:30:00.000Z""^^<http://www.w3.org/2001/XMLSchema#datetime> .
_:B3105a27f0059867877fb28653f5a6abc <http://def.isotc211.org/iso19156/2011/Observation#OM_Observation.result> _:B2598b988faa2635d5dd520f59f376b8e .
_:B3105a27f0059867877fb28653f5a6abc <http://www.w3.org/ns/sosa/madeBySensor> <urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j> .
<https://iow.smartdataspace.beta-vlaanderen.be/water-quality-observations> <https://w3id.org/tree#member> <urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn/2022-11-09T20:30:00.000Z> .
_:B727a9b2b6a6311cc83b677d439f4cf68 <http://def.isotc211.org/iso19103/2005/UnitsOfMeasure#Measure.value> _:B9776c69148d2ef600aed62aa2c85bd40 .
_:B2598b988faa2635d5dd520f59f376b8e <http://def.isotc211.org/iso19103/2005/UnitsOfMeasure#Measure.value> _:B436b5950c7725d6d9a38aaf6ca88b00f .
_:B9776c69148d2ef600aed62aa2c85bd40 <https://schema.org/value> ""1120""^^<http://www.w3.org/2001/XMLSchema#integer> .
_:B436b5950c7725d6d9a38aaf6ca88b00f <https://schema.org/value> ""673""^^<http://www.w3.org/2001/XMLSchema#integer> .Next in the Apache NiFi flow is a version materialization component. ""Version materialization"" refers to the process of removing version information from an LDES member and reverting it back to a ""state"" object, which only reflects the current state of the LDES member without historical information about changes.All information about previous changes is removed when you perform version materialization on an LDES member. Version materialization is needed since a consumer doesn’t want to store these versions in a database.The data is first converted to JSON-LD to easily convert it to a tabular structure. This JSON-LD file looks like this:{
 ""@id"" : ""urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-g9kEXxUeP28TNc6wJvFsau"",
 ""@type"" : ""Observatieverzameling"",
 ""Bemonsteringsobjectverzameling.lid"" : [ {
   ""@id"" : ""_:b8"",
   ""@type"" : ""Meting"",
   ""Observatie.geobserveerdKenmerk"" : ""https://data.vmm.be/concept/waterkwaliteitparameter/temperatuur"",
   ""Observatie:.phenomenonTime"" : {
     ""@type"" : ""http://www.w3.org/2001/XMLSchema#datetime"",
     ""@value"" : ""2022-11-09T19:30:00.000Z""
   },
   ""Observatie.resultaat"" : {
     ""@id"" : ""_:b6"",
     ""Maat.maat"" : {
       ""@id"" : ""_:b1"",
       ""https://schema.org/value"" : {
         ""@type"" : ""http://www.w3.org/2001/XMLSchema#double"",
         ""@value"" : ""1.153E1""
       }
     }
   },
   ""Observatie.uitgevoerdMetSensor"" : ""urn:ngsi-v2:cot-imec-be:Device:dwg-iow-8JgF3vxoYKoXzwWGUiX3Yc""
 }, {
   ""@id"" : ""_:b3"",
   ""@type"" : ""Meting"",
   ""Observatie.geobserveerdKenmerk"" : ""https://data.vmm.be/concept/waterkwaliteitparameter/conductiviteit"",
   ""Observatie:.phenomenonTime"" : {
     ""@type"" : ""http://www.w3.org/2001/XMLSchema#datetime"",
     ""@value"" : ""2022-11-09T19:30:00.000Z""
   },
   ""Observatie.resultaat"" : {
     ""@id"" : ""_:b4"",
     ""Maat.maat"" : {
       ""@id"" : ""_:b5"",
       ""https://schema.org/value"" : {
         ""@type"" : ""http://www.w3.org/2001/XMLSchema#integer"",
         ""@value"" : ""920""
       }
     }
   },
   ""Observatie.uitgevoerdMetSensor"" : ""urn:ngsi-v2:cot-imec-be:Device:dwg-iow-8JgF3vxoYKoXzwWGUiX3Yc""
 }, {
   ""@id"" : ""_:b7"",
   ""@type"" : ""Meting"",
   ""Observatie.geobserveerdKenmerk"" : ""https://data.vmm.be/concept/observatieparameter/hydrostatische-druk"",
   ""Observatie:.phenomenonTime"" : {
     ""@type"" : ""http://www.w3.org/2001/XMLSchema#datetime"",
     ""@value"" : ""2022-11-09T19:30:00.000Z""
   },
   ""Observatie.resultaat"" : {
     ""@id"" : ""_:b2"",
     ""Maat.maat"" : {
       ""@id"" : ""_:b0"",
       ""https://schema.org/value"" : {
         ""@type"" : ""http://www.w3.org/2001/XMLSchema#integer"",
         ""@value"" : ""11133""
       }
     }
   },
   ""Observatie.uitgevoerdMetSensor"" : ""urn:ngsi-v2:cot-imec-be:Device:dwg-iow-8JgF3vxoYKoXzwWGUiX3Yc""
 } ],
 ""http://www.w3.org/ns/prov#generatedAtTime"" : {
   ""@type"" : ""http://www.w3.org/2001/XMLSchema#dateTime"",
   ""@value"" : ""2022-11-09T19:30:00.000Z""
 },
 ""http://www.w3.org/ns/sosa/hasFeatureOfInterest"" : ""spt-00027-06""
}A JOLT transformation filters the relevant parameters and puts them in a structured JSON file (see output underneath).{
   ""id"": ""urn:ngsi-v2:cot-imec-be:WaterQualityObserved:dwg-iow-9NQCQNb4dJZ5J8kTACzdVn"",
   ""temperature_value"": ""1.152E1"",
   ""temperature_date"": ""2022-11-09T20:30:00.000Z"",
   ""temperature_sensor"": ""urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j"",
   ""conductivity_value"": ""1120"",
   ""conductivity_date"": ""2022-11-09T20:30:00.000Z"",
   ""conductivity_sensor"": ""urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j"",
   ""hydro_pressure_value"": ""673"",
   ""hydro_pressure_date"": ""2022-11-09T20:30:00.000Z"",
   ""hydro_pressure_sensor"": ""urn:ngsi-v2:cot-imec-be:Device:dwg-iow-csHTUVdGuPYK89L34yi88j""
}After this transformation, the data can be written to a PostgreSQL or Timescale database. The analysis you can do with data in Timescale includes but is not limited to time-series visualization, anomaly detection, forecasting, and aggregation of time-series data over various time intervals.Full CodeTo replicate the data flow in this article, please go to theLDES2TimescaleDB GitHub repository. It describes how to set up the dockerized Timescale and Apache NiFi, after which the data flow can be started using the supplied Apache NiFi setup file.Wrapping UpA Linked Data Event Stream is the core API of fast and slow-moving data. It is a data event stream of a group of immutable objects described as machine-readable RDF (such as sensor observations, address registers, or financial data).To write a LDES to a PostgreSQL or Timescale database, a data conversion flow is configured in Apache NiFi. Using this article as a guideline, you should be in good shape to write Linked Data Event Streams to a Timescale database.Contributors to this article areddvlanck (Dwight Van Lancker) (github.com),sandervd (Sander Van Dooren) (github.com)atFlanders Smart Data Space(Digital Flanders, Belgium). In a rapidly changing society, governments need to be more agile and resilient than ever. Digital Flanders realizes and supervises digital transformation projects for Flemish and local governments.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/the-power-of-linked-data-event-streams-and-timescaledb-for-real-time-management-of-time-series-data/
2022-11-16T14:00:00.000Z,"Expanding the Boundaries of PostgreSQL: Announcing a Bottomless, Consumption-Based Object Storage Layer Built on Amazon S3","We are excited to announce the initial launch, in private beta, of our new consumption-based, low-cost object storage layer in Timescale. This new capability expands the boundaries of traditional databases, allowing you to transparently tier your data across disk and Amazon S3 while accessing it as if it all lived in one single continuous PostgreSQL table. This means that you can now store an infinite amount of data in Timescale, paying only for what you store. Bottomless cloud storage for time series, events, and analytics is just one piece of our vision to empower you with exceptional data infrastructure so that you can build the next wave of computing.We started Timescale five years ago with a mission: to help developers build the next wave of computing through applications that leverage time-series and real-time analytical data.This mission led us to build TimescaleDB, a time-series database that gives PostgreSQL the performance boost it needs to handle relentless streams of time-series data at scale.On top of scalability and performance, another key concern for developers managing data at scale iscost efficiency. Time-series data is often collected at high frequency or across long time horizons. This scale is often a fundamental part of applications: it’s storing metrics about all IoT devices in a fleet, all the events in a gaming application, or tick data about many financial instruments. But this data adds up over time, often leading to difficult trade-offs about which data to store and for how long.To address this problem, we’ve developed several database features at Timescale aimed at making it easier for developers to manage their time-series data—likenative columnar compression,downsampling,data retention policies, anduser-defined actions. And indeed, these offer massive savings in practice. By compressing data by 95 percent, Timescale ends up much more cost-effective than vanilla storage options like Amazon RDS for PostgreSQL.Today we’re excited to announce how we’re extending this vision to a cloud-native future and building Timescale to supercharge PostgreSQL for time series, events, and analytics at greater scale and lower cost.Timescale now offers consumption-based, low-cost object storage built on Amazon S3.This new storage layer gives you, the developer, more tools to build applications that scale more efficiently while reducing costs. Leveraging a cost-efficient storage layer like Amazon S3 removes the need to pre-allocate—and pay for—an upper bound of your storage. When you tier data on Timescale, you will only pay for what you actually store while retaining the flexibility tokeep a limitless amount of data,and without being charged extra per query.This consumption-based pricing is not only transparent butan order of magnitude cheaperthan our standard disk-based storage. And what’s more, you can access this affordable object storage layer seamlessly from your Timescale database, meaning no need to create a custom pipeline to archive and reload data. All you’ll need is a single SQL command to automatically tier data based on its age, as suited to your application’s needs:# Create a tiering policy for data older than two weeks
SELECT add_tiering_policy ('metrics', INTERVAL '2 weeks');But why stop at cost efficiency? At Timescale, we strive to create aseamless developer experiencefor every feature we release. That means doing the heavy technical lifting under the covers while you continue interacting with your data in the simplest way possible.When applied to cost-saving object storage in Timescale, this means that even when data is tiered, you can continue to query it from within the database via standard SQL, just like you do in TimescaleDB and PostgreSQL. Predicates, filters, JOINs, CTEs, windowing, andhyperfunctionsall work! Reading data directly from tiered object storage only adds a few tens of milliseconds of latency—and this cost goes away for larger scans.We’ve natively architected Timescale databases to support tables (hypertables) that can transparently stretch across multiple storage layers. The object store is thus an integral part of your cloud database rather than just an archive.Here’s an example of theEXPLAINplan for a query that fetches data from disk and object storage (notice theForeign Scan):EXPLAIN SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;


QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
                -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text)The ability to keep your regular and tiered data both accessible via SQL helps you avoid the silos and application-level patchwork that come from operating a separate data warehouse or data lake. It will also help you escape the operational work and extra costs of integrating yet another tool into your data architecture.Starting today, tiering your data to object storageis available for testing in private beta for all Timescale users.Sign up for Timescaleand navigate to the Operations screen, pictured below, to request access. Timescale is free for 30 days, no credit card required.You can request access to our private beta via the Timescale UIBut, this is just the beginning.We plan to further improve object storage in Timescale to not only serve as bottomless, cost-efficient storage but also as ashared storage layerthat makes it dramatically easier for developers to share data across their entire fleet of databases.This is a huge step forward in our vision to build a data infrastructure that extends beyond the boundaries of a traditional database: combining the flexibility of a serverless platform with all the performance, stability, and transparency of PostgreSQL that developers know and love. Not just a managed database in the cloud, but a true “database cloud” to help developers build the next wave of computing.So yes, we are just getting started.✨ A huge “thank you” to the team of Timescale engineers that made this feature possible, with special mention to Gayathri Ayyappan, Sam Gichohi, Vineetha Kamath, and Ildar Musin.To learn more about Timescale’s new data tiering functionality, how it redefines traditional cloud databases, and how it can help you build scalable applications more cost-efficiently, keep reading.Bottomless Storage for PostgreSQLHaving native access to a cloud-native object store means you can now store an infinite amount of data, paying only for what you store. You no longer have to manually archive data to Amazon S3 to save on storage costs, nor import this data into a data warehouse or other tools for historical data analysis. Timescale’s new data tiering feature moves the data transparently to the object store and keeps it available to the Timescale database at all times.To enable this new functionality in PostgreSQL, we built new database internal capabilities and external subsystems. Datachunks(segments of data related by time) that comprise a tiered hypertable now stretch across standard storage and object storage. We also optimized our data format for each layer: block storage starts in uncompressed row-based format and can be converted to Timescale’snative compressed columnar format.On top of that, all object storage is in a compressed columnar format well-suited for Amazon S3 (more specifically,Apache Parquet). This allows developers more options to take advantage of the best data storage type during different stages of their data life cycle.Once a data tiering policy is enabled, chunks stored in our native internal database format are asynchronously migrated into Parquet format and stored in S3 based on their age (although they remain fully accessible throughout the tiering process). A single SQL query will pull data from the disk storage, object storage, or both as needed, but we implemented various query optimizations to limit what needs to be read from S3 to resolve the query.We perform “chunk exclusion” to avoid processing chunks falling outside the query’s time window. Further, the database doesn’t need to read the entire object from S3, even for selected chunks, as it stores various metadata to build a “map” of row groups and columnar offsets within the object. The result? It minimizes the amount of data to be processed, even within a single S3 object that has to be fetched to answer queries properly.Cost-Effective ScalabilityTimescale’s new object storage layer doesn’t just give PostgreSQL bottomless storage but also gives you, the developer, more tools to build applications that scale cost-efficiently.By leveraging Amazon S3, you no longer have to pre-allocate (and pay for) an upper bound of your storage.While Timescale already offers disk auto-scaling, your allocation is still “bumped up” between predefined levels: from 50 GB to 75 GB to 100 GB, from 5 TB to 6 TB to 7 TB, etc. Our new object storage layer scales effortlessly with your data, and you only pay for what you store.These storage savings can be meaningful: an order of magnitude cheaper than employing standard disk-based storage like EBS.So why isn’t this standard for all databases? We build solutions focused on analytical and time-series data. We are doing these transparent optimizations at the larger chunk level rather than the much smaller database page level. This way, we can effectively make the most of S3, which is optimized—for both price and performance—for larger objects. This same approach wouldn’t be practical when employing traditional page-based strategies for database storage.It’s Still Just PostgreSQL, But BetterAs we say,Timescale supercharges PostgreSQL for time series and analytics. But it’s always been important for us to maintain the full PostgreSQL experience, which developers trust and love. This is why we built TimescaleDB as an “extension” of PostgreSQL (although that “extension” has certainly gotten bigger and bigger over the years!).In our books, a smooth developer experience means that developers can continue interacting with all their data as if it’s a standard table—we do the heavy technical lifting under the covers. It should be invisible, and the more our improvements fade into the background, the better.Developers don’t realize that hypertables are actually heavily partitioned data tables—with thousands of such partitions—they just treat them like standard tables. Developers don’t see Timescale’sreal-time aggregationscombining incrementally pre-aggregated data with the latest raw table data to provide them with up-to-date results every time. They are meant to “just work.”We titled our 2017 launch post“When Boring is Awesome: Building a Scalable Time-Series Database on PostgreSQL.”We still strive to make Timescale seem “boring” to developers—simple, fast, scalable, reliable, and cost-effective so that developers can focus their precious time and minds on building applications.This focus on the developer experience similarly motivated our design of transparent data tiering. When data is tiered, you can continue to query tiered data from within the database via standard SQL—predicates and filters, JOINs, CTEs, windowing, andhyperfunctionsall just work.And what’s more, your SQL query will pull relevant data from wherever it is located: disk storage, object storage, or both, as needed, without you having to specify anything in the query.Here’s what it would look like working with relational and time-series data in Timescale, including tiered data. This example shows the use of sensor data, as you might have for IoT, building management, manufacturing, or the like. After creating tables and hypertables forsites,devices, andmetrics, respectively, you use a single commandadd_tiering_policyto set up a policy that automatically tiers data older than two weeks to low-cost object storage.# Create relational metadata tables, including GPS coordinates 
# and FK constraints that place devices at specific sites
CREATE TABLE sites (id integer primary key, name text, location geography(point)); 
CREATE TABLE devices (id integer primary key, site_id integer references sites (id), description text);

# Create a Timescale hypertable
CREATE TABLE metrics (ts timestamp, device_id integer, value float);
SELECT create_hypertable ('metrics', 'ts');

# Create a tiering policy for data older than two weeks
SELECT add_tiering_policy ('metrics', INTERVAL '2 weeks');Now, after you’ve inserted data into your metrics hypertable (and other relational data devices and site information into your relational tables), you can query it as usual. Reading tiered data only adds an extra latency of around tens of milliseconds, and this latency cost may even go away for larger scans.The following SQL query returns the maximum value recorded per device, per day, for a specific site—a fairly standard monitoring use case. Rather than showing the data results (which will just look normal!), we’ll show the output ofEXPLAIN.This command allows developers to see the actual query plan that will be executed by the database, which in this case includes aForeign Scanwhen the database is accessing data from S3. (With our demo data, three chunks remain in standard storage, while five chunks are tiered onto S3.)EXPLAIN SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;

QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
                -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text)Replace Your Siloed Database and Data WarehouseTimescale’s new data tiering functionality expands the boundaries of a traditional cloud database to incorporate features typically attributed to data warehouses or data lakes.The ability to tier data to Amazon S3 within Timescale saves you the manual work of building and integrating up a custom system or operating a separate data store (e.g., Snowflake) for your archival of historical data. Instead of setting up, maintaining, and operating a separate system alongside your production database (and a separate ETL process), you can simply work with a Timescale hypertable that serves your entire data lifecycle, where data is distributed across different storage layers.As we’ve illustrated, you can query regular and tiered data seamlessly from this table and also JOIN it to the rest of your tables, avoiding silos without adding more complexity to your data stack. This not only simplifies operations but also billing: unlike regular data warehousing systems (which typically charge per query, making it very difficult to forecast the final cost), in Timescale you’ll pay only for what you store, keeping your pricing transparent at all times.Request Access to Data Tiering TodayIf you’re already using Timescale,you can test data tiering today by requesting access to our private beta. We welcome your feedback to improve the product and better serve the needs of developers.To start testing data tiering in Timescale today:Sign up to Timescale.The first 30 days are completely free (no credit card required).Log in to the Timescale UI. In your Service screen, navigate to Operations > Data Tiering. Click on the “Request Access” button, and we’ll be in touch soon with the next steps.Bottomless cloud storage for time series, events, and analytics is just one piece of our vision to empower you, the developer, with exceptional data infrastructure so that you can build the next wave of computing.We plan to further improve object storage in Timescale to not only serve as bottomless, cost-efficient storage but also as ashared storage layerthat makes it dramatically easier for developers to share data across their entire fleet of databases.If that sounds interesting to you, please request access to the private beta and let us know!We’re just getting started!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/expanding-the-boundaries-of-postgresql-announcing-a-bottomless-consumption-based-object-storage-layer-built-on-amazon-s3/
2023-10-05T12:50:57.000Z,What InfluxDB Got Wrong,"There’s been a lot of talk about InfluxDB recentlyin the context of InfluxDB 3.0. While some of the commentary has been focused on technology (if there is one thing that creates hype it’s rewriting in Rust), a larger part has focused on issues with the company's (InfluxData) trajectory over the years. ‌‌“We're migrating off of InfluxDB due to that rollercoaster, honestly. It's hard enough to find time to maintain the monitoring stack at work. Casually dropping ""Oh, and now you get to rebuild the entire Grafana to change the query language"" on that doesn't help. And apparently, version 3 does the same thing, except backwards.” (source)‌‌“Same here. I joined my current company 3 years ago when Influx v2 was coming out. I was supposed to build some analytics on top of it. It was very painful. Flux compiler was often giving internal errors, docs were unclear, and it was hard to write any a bit more complicated code. The dash is subpar to Grafana but Grafana had just raw support. There was no query builder for Flux so I tried building dashboards in Influxv2 but the whole experience was excruciating. I still have an issue open where they have an internal function incorrectly written in their own Flux code, and I provided the fix and what was the issue, but it was never addressed. Often times I had a feeling that I found bugs in situations that were so basic that it felt like I was the only person on the planet writing Flux code.” (source)‌‌“We are Influxdb enterprise customers and looking to do the same thing. They've kept their enterprise offering on 1.x, which has kept us mostly happy, but seeing what's going on in their OSS stuff is horrifying, and we're looking to avoid the crash and burn at the end of the tunnel.” (source)We’re Timescale (the creators of TimescaleDB) and we also compete in the time-series market, so we are undeniably biased when we are talking about InfluxDB as a piece of technology (although we always try to make our benchmarks as balanced as possible). But as developers, it also saddens us to see popular projects lose momentum. InfluxDB achieved something remarkable: building any company around a popular open-source project is not easy.The unfortunate thing is that, somewhere along the way, InfluxData has squandered much of the developer goodwill they’ve put so much hard work to earn. Developers who use them for time-series, IoT, and observability workloads grew increasingly frustrated, and as a company, they left opportunities open for other projects to capitalize on their mistakes.As developers of our own database company, we have to learn from other companies that build on open-source. So we asked ourselves: how did this happen? What did InfluxData get wrong?What InfluxDB Got WrongInstead of maturing their database, InfluxDB did not one but three backend rewritesIn the world of databases, performance is essential, but it’s not all that matters. At the end of the day, the reality of running a database in production implies that stability is key for developers. You want your database to “just work” and be as easy as possible to build on top of it. You need it to keep those qualities over a long horizon. If your database changes how things work, that translates to technical debt, which needs to be paid down before you can upgrade.So far, InfluxDB has been completely built from stratch not once but three times, from 1.x to 2.x to now 3.x. None of these were backward compatible. All of them were a “bold new approach” which promised to solve all of the problems developers faced.The recurring rewrites of InfluxDB have put its user base in a precarious position. Each version iteration not only demanded so much time and effort in order to migrate; it also challenged the trust that developers had placed in the initial promises of InfluxDB as a product. It seemed as if the allure of creating something new and innovative overshadowed the fundamental task of maintaining and refining existing products with an actual user base.‌‌‌‌We get it: focusing on keeping things stable while building foundational operational components is not as exciting as talking about full database rewrites. But the picture looks quite different to the developers building their application on top of a database.The design instability of InfluxDB has not only been damaging for their user base (and for the company’s credibility), but it also has very natural consequences regarding the reliability of the database.Since InfluxDB has been built from scratch (and more than once), it had to implement its full suite of fault-tolerance mechanisms (e.g., replication, high availability, backup/restore) and on-disk reliability (e.g., to ensure all its data structures are both durable and resist data corruption across failures).This is a daunting task. Some of these capabilities are, in fact, either still lacking in InfluxDB orconfined to the Enterprise version of the product. But even once they’re done building these, these capabilities have to be battle-tested.Getting all the corner cases right when building a database is extremely hard: every database goes through a period when things get perfected from real-world experience. The big advantage of PostgreSQL is that it went through this period in the 1990s, while InfluxDB is still figuring things out today.Just to be clear, we don’t think technological innovation is wrong. But you can’t reasonably expect your users to adopt three completely different solutions in a short period of time.InfluxDB also changed their query API three timesThe three major versions of Influx also came with different query languages.Via InfluxQL, a SQL-ish query language,InfluxDB 1.xwas betting on creating a “middle ground,” an environment familiar enough yet tailored for the specialized needs of time-series data. WithInfluxDB 2.x, they pivoted to Flux, which was a massive paradigm shift. Users were now tasked with learning a new and proprietary query language and adapting their entire codebase to it if they wanted to migrate with InfluxDB 2.x. Now,InfluxDB 3.xis getting back to InfluxQL, frustrating the same developers who believed their promises and made a huge effort to migrate.And this is not a problem you can fix by throwing money at it—not even cloud customers are safe from the back-and-forth language changes. Adding insult to injury, Influx Cloud 1.x runs InfluxDB 2.x (Flux), and Influx Cloud 2.x runs InfluxDB 3.x (InfluxQL). The promise of leaving infrastructure hassle behind wanes in the face of a brand-new and challenging onboarding process.To further confuse matters, they also now support the DataFusionSQLvariant usingFlightSQLas transport, allegedly also supporting connections via anyPostgreSQL-compatible client—except we tested that and it doesn’t work. The reply from Influx support was this: “At this point, you can query InfluxDB IOx using FlightSQL plugin and supporting the Postgres wire protocol has been stopped.”Nobody has the time to learn a new query language, build new connectors, put together new dashboards, and rewrite application code every two years. Database maintenance is already hard enough. Your database vendor should be taking work off your hands, not making it worse. Every hour spent working on your database is taking off from the core objective of the developer, which is building, running, and growing an application. To understand this is to respect the developer’s time and effort.This is not only a theoretical concept but a foundational design principle that InfluxData seemed to miss.InfluxData’s lack of focus confused their users (and hurt their market share)InfluxData started with a great project (InfluxDB 1.x, a time-series database). Soon enough, their focus seemed to scatter, firstby building the TICK stack(which was close to being an observability platform) and then with InfluxDB 2.x, with which they seemed to double down on prioritizing the platform vs. the database. Now, with Influx IoX (a.k.a. Influx 3.0), this has flipped once again, and they’re doubling down on the core database to abandon the platform they had built.This ambiguous market positioning opened up the door for other, more focused solutions to emerge, such as Prometheus and Grafana, which ended up dominating the metrics and monitoring space.It also didn’t help that the TICK stack, as it was originally conceived, was also sort of abandoned in the InfluxDB 2.0 rewrite process.Developers who had invested significant time and resources into integrating these tools into their systems were now told to migrate to something new for reasons that were not completely clear nor justified. It felt as if Influx kept busy looking for product-market fit in other places while forgetting about the users they already had.Perhaps if InfluxData had decisively committed itself to the metrics and monitoring use case when they had the market advantage, it could have focused its core engineering resources to carve out a definitive niche in the sector. And if Influx had decided to focus on improving and maturing a database that was already quite great, TimescaleDB would probably not exist today (more below).Now, it’s time for us to be fair. We have also been guilty of a lack of focus at times. In 2020, we builtPromscale, our own observability tool built on TimescaleDB,to then deprecate it earlier this year. After reevaluating our company priorities, we realized our mistake: we were not an observability company; there were great open-source solutions thriving in that space already; and most importantly, by dedicating core engineering efforts to Promscale, we were moving away from what we knew how to do best, which was helping developers build better applications.There was one big difference here though: Promscale never made it to version 1.0 or out of beta.InfluxDB users got penalized with too many product optionsThere’s something to be said about the value of simple choices. Databases are very complex pieces of software, and keeping things easy is not always possible, but as database companies, there’s a lot we can do to simplify decision-making for our users.Simplicity as a value does not translate for InfluxData, starting with their product portfolio. As the company oscillated between roles and projects, never quite committing to a definitive path, they discharged the cognitive load onto the developer, who now has to navigate betweenInfluxDB OpenSource(which exists on its1.x,2.x,and3.xversions),InfluxDB Cloud(which exists at 1.0 running Influx 1.0, and 2.0 running Influx 3.0),InfluxDB Cloud Serverless,InfluxDB Cloud Dedicated,InfluxDB Clustered,InfluxDB Cloud,InfluxDB IoX,InfluxDB Enterprise,InfluxDB Edge, etc. We might be missing some.This is without even beginning to dive into the rabbit hole of trying to work out which versions (or features) are commercial and which are open-source.InfluxDB is not PostgreSQLYou’re probably thinking, “Of course Timescale would say that,” but when you think about it, the particular challenges of time-series workloads gave birth to InfluxDB as a specialized technology based on the premise that time-series data was too much to handle for relational databases.The thing is that developers don’t want to use niche databases: they want to use PostgreSQL. Why specialize when you can generalize?Databases are not just tools but entire ecosystems with distinct query languages, interfaces, and operational protocols. Filling your stack with niche databases means that you’ll end up spending so much time battling with new technologies that are still somewhat untested. And your data will be siloed and locked between different places. Joins won’t be possible, new use cases will rely on support in the database stacks, you’ll have trouble with technology not integrating with your database, and your operational overhead will multiply.Building on PostgreSQL simplifies a developer's life. SQL’s universal acceptance ensures immediate productivity. You can join your time series data with any other table. You can run complex analytical queries. You can tap into a robust and reliable ecosystem with easy integrations and a wealth of ready-to-use tools and extensions. This environment is enriched by a global community, guaranteeing continuous support. Postgres isn't just another choice; it's a smart strategic move.Some Things That InfluxDB Got RightDespite the lengthy discussions on what InfluxData (the company) got wrong, there is one thing they got right, and that’s the original InfluxDB product. It’s only fair that we take some time to point out the area where it excels (even if you might have to traverse different versions, or query languages, to get there).InfluxDB is good for IoT workloads where you’re ingesting millions of time series per second, with multiple labels and values per item(a wide-table schema). You want to read that data back or analyze each time series individually, that’s about where your workload ends.Apart from the database,Telegrafdeserves a distinctive mention—its a fantastic tool that survived the Influx 2.x odyssey. Kudos to the InfluxData team for keeping it alive.We also love the time to awesome concept. At least in the world before 3.x (and specially with 1.x), it was quite easy to get InfluxDB up and running. It's hard to build a data product that allows developers to realize value quickly, and Influx managed to do that. This is something that we take as inspiration in Timescale and that we're working to improve.Final ReflectionBefore we end up, we happen to have a personal story with InfluxDB that goes years back.Timescale first started as an IoT company, storing data from over 100,000 devices. We needed a place to store all this sensor data and picked an off-the-shelf time-series database to do so alongside our main PostgreSQL database.But our experience was not great. We hit performance issues, we hit stability issues, we hit query language issues. Early on, we wanted to display metadata for all the online devices. But our data was now siloed: the device sensor data lived in the time-series database, and the device metadata lived in PostgreSQL. So even that simple request: “Show me all devices that are online right now,” which really should have just been a simple JOIN between two tables, required glue code and an engineering sprint to get done.When we met with the time-series database company to talk about our pain points and see the future roadmap for the database, we learned that their plan was actually to build a complex stack that included data processing and visualization—they were essentially putting aside the well known, well-documented problems their database had. We didn’t need any of that other stuff. We just wanted a better database for time-series data.We left that meeting with the realization that we had no choice but to build that better time-series database ourselves. We ended up building it as an extension of PostgreSQL because we believed (and still do today) that data points don’t exist in a vacuum; they need to be joined and enriched with other data, and there’s an entire developer workflow and ecosystem of tools around those data points that need to be considered. That small idea eventually became our main product and the foundation for a completely new company.In case it’s not obvious, that time-series database was InfluxDB. That conversation happened in May 2015. And if InfluxData had made better product and business decisions, Timescale would not exist today.Throughout the years, we have made plenty of our own mistakes. The point of this post is not to point fingers or pretend that database companies are not allowed to make mistakes, pivot, or explore new ideas. It’s about the importance of listening to your developers and learning from past experiences. There’s a lot to be learned from the InfluxDB story, lessons that we’re reflecting on at Timescale. We’ve made some mistakes, but we continue to do better.Wrap-UpAs these database discussions come and go, PostgreSQL's popularity continues to grow. We remain convinced that PostgreSQL is the bedrock on which modern applications can and will be built. Its proven reliability, versatility, rich ecosystem, and the power of SQL as a query language is a very hard-to-beat combination by any other emerging specialized database.And Timescale is the missing ingredient that makes PostgreSQL ready for your time-series data workloads.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/what-influxdb-got-wrong/
2023-01-18T16:23:18.000Z,What's New in TimescaleDB 2.9?,"The latest version of TimescaleDB 2.9 is now available onTimescaleand as an upgrade option foron-premise installations. This major release includes the following new features:Hierarchical continuous aggregates:you can now create and use a continuous aggregate on top of another continuous aggregate.Background jobs:introducing fixed schedules for background jobs and the ability to check and troubleshoot job errors.Hyperfunction improvements:improvedtime_bucket_gapfillfunction allows to specify the time zone to a bucket.Multi-node improvements:use ofalter_data_node()to change the data node configuration. With this function, you can now configure the availability of the data node.Let’s explore these improvements in more detail.Welcome, Hierarchical Continuous AggregatesWith TimescaleDB 2.9 and its later versions, users can downsample their time-series data more efficiently and faster than before by creating continuous aggregate hierarchies.Eachhierarchical continuous aggregateis defined on top of the previous one, providing a lower granularity view of the same dataset while making the refresh process for higher-level continuous aggregates lightning-fast. This also enables TimescaleDB users to define more diverse sets of continuous aggregates without significant additional costs.For example, a continuous aggregate providing a daily summary only needs to go through 24 records each time it is refreshed when built on top of a continuous aggregate that provides an hourly summary—forget those tens of thousands of records stored in the raw hypertable for that day.Imagine that you have a finance dashboard, as shown in the example below. With hierarchical continuous aggregates, you can visualize your stock status by week by using pre-aggregated daily data (without going through raw data in the hypertable). This results in super fast response times, which prevent a laggy user experience of the dashboard.Diagram example of hierarchical continuous aggregates functionality for a finance use caseFor a more in-depth read about hierarchical continuous aggregates, their examples, and code snippets, read Chris Englebert’s blog post “An Incremental Materialized View on Steroids: How We Made Continuous Aggregates Even Better.”More Scheduling Options and Better Visibility for Policies and User-defined ActionsTimescaleDB natively includes support for automation policies, such as the following:Continuous aggregate policiesto automatically refresh continuous aggregatesCompression policiesto compress historical dataRetention policiesto drop historical dataReordering policiesto reorder data within chunksThose policies, together with user-defined actions, are meant to run at regular intervals. At a high level, they are jobs scheduled to run asynchronously in the background, facilitated by our job scheduler.With TimescaleDB 2.9, you can now set jobs to run at specific times by using our newly introduced fixed scheduling semantics. For example,by setting a continuous aggregate policyto start at03:00with a 24-hour schedule interval, the job will always be executed at 3:00 a.m. every day.We also introduced better visibility by providing information aboutruntime errors encountered by jobsrun by the automation framework.Other Improvements in TimescaleDBWe also continued to improve thetime_bucket_gapfillfunction to allow specifying the time zone to a bucket and introduced multi-node improvements. To be more precise, we improved the use of thealter_data_node()function by introducing the option to configure the availability of the data node.Try TimescaleDB 2.9If you are using Timescale, upgrades are automatic, and you’ll be upgraded to TimescaleDB 2.9.1 in your next maintenance window.New to Timescale?Start a free 30-day trial, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/whats-new-in-timescaledb-2-9/
2022-12-06T14:00:48.000Z,How to Design a Better Developer Experience for Time-Series Data: Our Journey With Timescale's UI,"Part of the Timescale mission is to create a seamless developer experience for those working with time-series data—something that user experience and user interface (UX/UI) design play a massive role in. With Timescale, we're building a cloud-native database platform from scratch. Behind it, there’s a multi-disciplinary team of designers, product managers, and engineers working closely together to build, iterate, and progressively improve our platform. (By the way, hi 👋, this is the Product Design Team speaking!)Designing the user interface of a modern database platform is a peculiar challenge. As with any great design, you need to prioritize the user (or, in this case, the developer) experience, optimizing for the wide and diverse range of workflows within the platform. Our customers build many things with Timescale, from powering their frontend dashboards to running complex analytics.Some database platforms are very complicated to manage, which we wish to avoid by making things as easy and seamless as possible for the user. Simultaneously, we aim to keep the control in the hands of the developer—we strive for transparency and openness, steering away from black boxes.Today, we’re sharing our perspective on how to solve these problems by taking you on a journey through the evolution of the Timescale Cloud UI. We’ll start right at the beginning, when our product was called Timescale Forge (remember those days?), all the way up to its current state, closing it off with a few lessons we learned along the way.P.S.When we wrote this blog post, the UI’s state was very recent 👀🔥 as we just launched an important UX redesign that took us many months to complete, which we dubbed Timescale 2.0.P.S.S.Since our Timescale 2.0 redesign, we have completely rebranded, including, of course, our UI. Our product name has also changed from Timescale Cloud to a cleaner Timescale, which is also our brand name. Check outour websitefor the changes, andread more about the rebranding in this blog post—we promise to share our design lessons from the rebranding sometime!But now, let us tell you the whole story (up to Timescale 2.0). It has been a fun rollercoaster teamwork effort, and we want to share it with you!Improving Developer Experience: Devising 2.0If you are not a Timescale user or don’t know us, let us give you a little context first (if you already do,feel free to jump to the next section).Timescaleis a modern, cloud-native relational database platform for time series and analytics built onPostgreSQL.If you’re not a developer and this sounded like advanced math, let’s take the engineering jargon down a notch: we are a cloud product that allows developers to ingest, store, and analyze time-series data. (Time-series data is a collection of repeated measurementsthat will enable you to track changes over time. You will find it in health apps that check your pulse, in the finance sector to monitor stock markets, or in smart home devices, for example.)Timescale is fast (with 350x faster queries and 44 % faster ingest than the competition) and simple to get started (you can spin up an instance in 30 seconds). Also, it offers advanced functionality to help store high volumes of time series data for less money (likecolumnar compressionordata tiering to S3). The product is built on top of PostgreSQL, one of the most reliable and loved open-source relational databases, and it runs in AWS.From a simplistic point of view, progressively improving the design of a database product like Timescale may look like a simple facelift. But, as we mentioned, developer experience is vital to us, so the entire design evolution process comprised more than visual changes to make it “prettier.” Our goal was to improve the user experience dramatically.So, we spent months planning, researching, understanding user paths and pain points, iterating, debating, iterating, testing, iterating, and drinking many cups of coffee (and tea!) before launching the massive Timescale 2.0 redesign.To correctly tackle a redesign, you can’t redesign for yourself according to your assumptions and sense of style. We did an in-depth analysis, and for every decision, we considered the user needs and our team’s needs, the user pain points, the user experience, the company goals, the brand, our value proposition, etc. We needed to be entirely sure about the decision to redesign our platform because it is a long and challenging—but also fun!—journey once you start!But, we didn't redesign the layouts just for a better developer experience. We hoped to build a solid foundation on both the design and development sides. We improved engineering velocity by switching all the design elements to reusable components. This means that some parts we created from scratch and others we re-examined completely: UI library, user flows, a storybook with all the components, and a new code.But why did we make this decision? What was wrong with the interface? How was the process? How long did it take to make the changes?Let’s go back in time to answer all these questions.Episode I: The Timescale Forge EraWhen we created Timescale, it had a different name: Timescale Forge. The product was started in 2019 when we had less than 30 people in the company, and of course, the UI Team was incredibly small. In the beginning, the Design Team comprised only one person, and the Frontend Team was only three (working on the product and everything related to Timescale), so all the work was admirable.Timescale Forge was the first minimum viable product (MVP) of this fantastic product we are still building, which we call Timescale Cloud (at the time of writing this blog post). But before swapping names, in early 2021, the company started to grow, so we created a plan to improve the interface, functions, user experience, and UI structure. We analyzed the interface and detected some problems:We were using a small piece of space on the screen to interact.We had tiny font sizes to fit that small space.The functions and user experiences needed to be more consistent.We made less-than-ideal choices in some of the interactions.The UI was stiff and old-school.This is how our interface looked then:0:00/1×Timescale Forge, March 2021Creating a service in the old Timescale ForgeAnd these are some of the problems we identified:A. There was a lot of dead space.B. We had an important (call to action) CTA with a small font size (10 pt).C. Some of the instructions looked like a banner.D. We had some fonts in size 10 pt.E. There were some nonsense actions (like clicking “Next” after completing your name).F. We used a lot of monospace fonts.G. We confused users by applying an orange-gold color (usually associated with warnings) to interactions.H. The warning color (brown) was very close to the main color.Our Design and Frontend teams were smaller then, so we knew we couldn’t handle all the external and internal improvements we wanted. So, we divided them into baby steps.The first step we made was to increase the screen space and make the font sizes bigger to optimize our users' experience and legibility.Being legible means ""that can be read."" Legibility is determined by respecting the visual design and typography rules (we didn’t have much then), such as good contrast, tracking, kerning, and leading.Some rules to have excellent legibility include the following:Use a reasonably large default font size: we used small font sizes, and the screen was hard to read. We had some text in 10 pt or even 8 pt.Create high contrast between characters and background: we changed some grays to darker grays to avoid low contrast.Use a clean typeface: our principal font is Inter, a clear and simple font. But we also use JetBrain (a monotype font). One of our changes to increase legibility was to define clear use cases for the monotype font instead of using it randomly because it can sometimes be hard to read.Based on these rules, we implemented a few changes to the Timescale Forge UI, which you can see in this video:0:00/1×A comparison of Timescale Forge between March 2021 and June 2021As you can see, we only fixed a few elements:A. Less dead spaceB. A larger CTA (14 pt) that’s disabled when it needs to be disabledC. No changesD. Increased font size from 10 pt to 14 ptE. Removal of nonsense actions (like clicking “Next” after completing the name)F. No changesG. No changesH. No changesWhile this small Forge redesign may not seem very powerful, low legibility made our product hard to use, and a product that is hard to use is not a good product. However, we were determined to make it even better, particularly as a name change loomed on the horizon.Episode II: From Timescale Forge to Timescale CloudAfter this minor update in July 2021, we changed the name of our platform from Timescale Forge to Timescale Cloud— of course, that meant we needed to revisit the UI.With the product renaming, we had to change the logo, so it felt like a fantastic opportunity to update the design. But we didn’t have the time to do a full-blown overhaul in just a couple of months before the announcement of the new platform.Still, the seed of a significant redesign sprouted at this moment. We knew the change should be fast, but we wanted to clear the path for the new interface. So, we did a UI exercise! Each designer scanned the interface for two to three things they wanted to change and then explored the different visual paths. The idea was to set a vision for the future through fun exploration so that we could find a base for the next iteration.Following this exploratory exercise, the team combined all their options and built something similar to the layout that we wanted to implement in the future. Then, we took a step back to make a streamlined update where we immediately applied these changes.The results? A new logo, colors, and minor changes made the interface softer and more inviting.The exercise also opened the door to discussions about the look and feel of the redesign, project timelines, and how it would affect the overall product.But before moving on to the actual redesign, check out our interface evolution! This is the before:Timescale Forge overview between March 2021 and June 2021The in-between phase, while we were working on our UI exercise:Timescale Cloud overview—UI exercise to start envisioning the futureAnd this is the after version:Timescale Cloud overview, September 20210:00/1×Episode III: The Rise of 2.0In September 2021, afterthe launch of the new Timescale Cloud, we found the perfect timing to officially start planning the next iteration, the so-called Timescale 2.0. It was all about our end goal: to create a better interface and developer experience for our users and a better structure for us. So we sat down—the Product Design, Product, and Engineering teams—and started planning everything.Let’s share what each of us had in mind.Design: It's all about the developer experienceOn the design side, we decided to create the following:1. A newdesign systemand UI libraryin Figma to have a guide with all the rules, specifications, and reusable components to build our interface.2. Redesign the interface from scratch.We kept some functionalities and changed others, but we redesigned the layout completely.3. Become fully user-centered.Instead of designing for the entire user journey, we had only focused on single screens because we started with a tiny team without the capacity to work full-time in the Product area. Now, we always deploy the flows taking into account the complete user path.4.Redesign the illustrations.Our company's mascot (Eon) is a tiger, and we decided to create a flat version and integrate it better into our product.Engineering: The right tools for your libraryAnd this is what we did on the development side:1. We built a frontend librarythat mapped to Figma as closely as possible. Each component either directly matched a design component or, where the underlying logic was the same, matched several components using styling variants.Having reusable components is a common development pattern these days, but implementing them well requires a solid maintenance plan. Developers need to be able to, at a minimum: know what components are available and how components function, make changes quickly and check whether changes to one component have unforeseen effects elsewhere.Creating mostly reusable components also forced us to be more diligent about separating display logic from app logic, allowing us to easily swap in new styles if we need to make changes.2. We moved toStorybook, a custom-made tool for organizing components. The idea was to have the same well-organized library for both teams but in different applications (Figma vs. Storybook). Using both, we can maintain the consistency of the product and save time creating and reviewing new flows.3.We adoptedChromatic,the cloud-based version of Storybook. The main benefit this brought us was another way to connect design and development. Developers can run Storybook locally while they build, but any proposed changes to a component in the library on the developer-side surfaces on the Chromatic site.Chromatic auto-builds the library for each change and highlights the differences between that component and any dependent components. This makes it easy to view and discuss changes between the two teams.We could have built something ourselves to host the Storybook library, but there’s always a trade-off between paying someone else to do a task and paying for development and maintenance time to do it yourself. In this case, the folks at Chromatic are the same folks that put out Storybook and are constantly improving their product, so we felt comfortable letting them do what they do best!4.We converted most of our code fromJavaScripttoTypeScript. This doesn’t directly affect the component design, but it gave us several benefits regarding work processes. TypeScript gives JavaScript static type checking, which generally reduces errors while passing variables between components (and functions in general). It also makes it easy to specify the format of arguments a function should receive. All of this allows us to build new things with more confidence.Our guiding principlesWe couldn’t have completed these tasks without a few guidelines, and two emerged as especially crucial for this project. The first was building a treasure trove of design and development elements to speed up the creation process—our libraries— and the second was embracing a fully user-centered design.Here’s why these matter to us.Why is it important to have a library?When you need to move fast, improve, and iterate constantly (as is common in startups), it is normal to have tons of inconsistencies throughout the product. It is hard to keep the correct balance between shipping fast and making it all perfect and consistent without any technical debt. That’s what happened to us.But if you have a robust design system and library, you can minimize and avoid inconsistencies. Another significant benefit is reducing code duplication. If a component exists and is easy to find, it prevents you from rebuilding that component somewhere else in the product. So, those files will become your source of truth with all of the product requirements to iterate and build new flows. You can adequately set up the text styling, colors, paddings, shadows, containers, distances, icons, illustrations, behaviors, motion, breakpoints, etc.Besides setting up all the rules, a library makes design and development work much faster because you already have the components built and can reuse them. So, you may ask, if we needed to move fast, why didn’t we have a library from the beginning? And, yes, that question makes sense. We all should have libraries to work better and faster from the start. But the truth is that creating a library takes a lot of time, so that’s why it took us so much to finally have a good library.But, basically, with a library, you have all the elements already built. If someone needs to update something, you need to edit the mother file (on both the design and development sides), and that change will impact all of your product screens, keeping up the consistency on top of everything. And consistency in a product is another key to a good product.Preview of the Timescale UI Library—FigmaWould you like to have a sneak peek at our UI Library? Click here to see aninteractive prototype.Preview of the Timescale StorybookAnother game-changer: Embrace user-centered designWhen we design a digital product, it is vital to understand who the user is and how they will use it. If we don’t include the user in the picture, it’s like you are watching only half of the movie.Having a user-centered design means that our processes put the user always as the first thing to discuss. We take into account user requirements, objectives, and desires. Satisfying users’ needs becomes a priority, and all the decisions are evaluated in the context of whether it delivers value to the users or not.First, we need to change how we think and, after that, change how we work. For example, we changed something as small as how we deploy the flows on Figma to embrace this new methodology. We used to design by screens/sections, and now we deploy the complete user flow (starting with the happy path) to understand the full picture, catch all the user corner cases and avoid dead-end roads with no solutions for the users. We are not masters of user-centered design yet, but we are starting to incorporate it into our daily workflow to improve our product.Episode IV: Ready to LaunchSince we began working on 2.0, we have gone through many iterations, prototypes, stakeholder meetings, updates, plans, estimates, designs, etc. More than one year may sound like a crazy timeline to work on a redesign, but we did it while improving our current interface. You can’t stop updating the product, releasing new features, and incorporating new elements just because you are working on a redesign.What you need is to change the car’s wheels while in motion. So, for more than one year, the team worked simultaneously on two paths: moving forward with Timescale 1.0 (that’s what we call it internally) and building the new features of Timescale 2.0.Several people asked us why we didn’t break such a massive project into smaller pieces to release it faster. We split the new proposal into MVPs and future iterations, but we wanted to make a big first release for two reasons. The first is technical incompatibilities. The development team built the new interface with an improved structure and code, making it impossible to release it all at once (so stay tuned because further improvements are coming soon!).And the other reason is focused on the developer experience: the new interface looks and feels different, and having both simultaneously would give users the idea that we’re not consistent in our product. Plus, some screens could look broken.But, after a year’s work, we finally launched 2.0, and we are ready to show it to you. 🥁0:00/1×This is how the Services page looks now in Timescale Cloud. This is the first screen you’ll see when you log in, where you can switch between your different Timescale databases.Timescale, Services page, December 2022When you click on your Services, you’ll land on the Overview page, which now lets you see a glimpse of the key info for your database configuration, service metrics, and pricing. We also made this screen actionable: if you want to resize your service, for example, you can click on the contextual menu (those three little dots) on the left of the storage card, for example, and simply add more storage to your service.Timescale Cloud overview, December 2022 (Did anyone catch The Office reference?)Another page we’d like to show you is the Explorer screen. This compiles information about your database internals—tables and hypertables, compression, continuous aggregates, etc.And in the Operations tab, you canseamlessly resize your database;configure storage autoscaling;add replicas for high availability and/or for scaling reads;configure VPC peering;upgrade your major Postgres version;request access to data tiering, and much more.So, first thoughts? We know we’re biased, but we’re very happy with what we achieved and hope our users will enjoy it as much as we do!Want to have a closer look at our interface?Browse our interactive prototype! Or, if you want to go beyond looks and get a taste of the frictionless developer experience we created for you while reaping all the benefits of a modern, cloud-native relational database platform for time series,you can try Timescale for free for 30 days (no credit card required).Epilogue: The Journey Has Just BegunAs with so many journeys, ours wasn’t over when we implemented the new redesign and structure—it had only begun. Now that we had everything in place, we started the last-mile work of creating a launch strategy, updating the documentation, testing the interface with internal users, training our internal teams, preparing material for the launch (like this blog post), etc.This is the first lesson we can share about such massive projects: our evolutionary journey has just started. We have to continue iterating, educating users and our colleagues, and promoting our new interface.But after more than a year’s work where we experienced all possible emotions and learned so much, there are a few more good lessons:Never underestimate work: realistic planning is key.Since starting this initiative, we have probably made a million estimates. Rebuilding everything takes time, especially when you are building a better structure and constantly improving the flows because you keep shipping new features. So, never underestimate your work!The dream team: product managers, engineers, and designers working together.It took us a couple of months to reach the desired level of teamwork for this project because it is hard to prioritize a redesign when you have new features to launch. But here’s the deal: great planning and teamwork are key to moving forward.You don’t need to complete your designs before starting to code. The Design Team waited until we had the UI library and all the flows’ drafts ready to make the official hand-off to the Engineering Team. Looking back, we probably could have saved some time and completed the hand-off while still working on the flows. But, it seemed right to do it that way because we were launching other features, and we didn’t want to “steal” development efforts when our flows weren’t yet closed.Break the process into manageable tasks.We did this from the beginning, which was the best way to go. We divided everything into manageable items on GitHub: flows, components, iterations, quality assurance, etc., and we split the assignments internally between the team. A concrete task is not as overwhelming and has a better chance of completion in less time.Learn when to say “yes” to feedback and when to say “let’s come back to it after the launch.”Receiving feedback is vital to any project because, with every input, the design grows stronger and more logical. More eyes are better than just a few. But you need to know when to stop receiving feedback, or you’ll never end the project. So learn when to ask for feedback and when to save issues for future iterations, focusing only on MVPs.And, speaking of feedback, we would be happy to hear your thoughts about Timescale 2.0! You can reach out on ourCommunity Slackand leave feedback so we can continue to improve our developer experience for all our users. And watch this space: as we said, our journey has just begun, and we have more UI news coming up soon!Haven’t tried Timescale yet and would like to get a look (and feel) of our new UI?Join us in a 30-day free trialfor a trip down faster queries, worry-free operations, and of course, a seamless developer experience.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-design-a-better-developer-experience-for-time-series-data-our-journey-with-timescales-cloud-ui/
2022-10-12T13:14:09.000Z,How to Test Performance Improvements on Up-To-Date Production Data in Timescale,"Testing performance improvements (or really anything related to your production database) can be a daunting task. Performance improvements, in particular, are especially challenging—you must test them first to ensure that the change produces a performance improvement. At a minimum, when changing something in our production database, we would like to make sure that our change doesn’t actually degrade performance before we apply these changes.Thankfully, this is one area in which operating a managed database in the cloud can help since we can leverage the flexibility of a cloud infrastructure to do performance testing (or testing of any kind) rapidly and safely.In this blog post, we’ll offer you some strategies to make testing easier from the perspective ofTimescale(30-day free trial, no credit card required!)—but even if you’re not a Timescale customer, we hope you find advice in this blog post that you can directly apply to your own environment.Performance Improvements: What to Look ForLet’s start from the beginning: what to look for if your database service is not performing as expected?Database performance is highly dependent on your use case—that’s why having direct access to a technical support team is so valuable. On a high level, we usually recommend our customers look at two areas when they see performance issues:Do you have enough CPU and memory?The first possibility is that you’re maxing out your database CPU or memory. One possible performance improvement to test would be changing the configuration of your service to see if, for example, adding more memory and CPU improves the performance.As we’ll see later, this is super convenient to test in Timescale using forks.Can your database configuration be improved?Another type of performance improvement may require configuring your database parameters more optimally. In the case of Timescale, you can start by following this advice:Timescale parameters you should know about (and tune) to maximize your performanceHow to optimize your ingest rate in a Timescale serviceTesting your chunk sizeIdentify slow queries with pg_stat_statementsUsing downsampling to fix slow queries in GrafanaThese potential improvements can also be tested using forks in Timescale.How to Run Tests Fast and Easy: Meet One-Click ForkingAlright: before we move on with our advice on testing for performance, let’s pause for one second. What are these “forks” that we keep talking about?Modern cloud databases give us one gift: flexibility. In Timescale particularly, there’s a way in which you can run all tests fast and easy:you can create copies of your production database in one click, changing their configuration as you please.These copies of your production database (which we call forks) will replicate your production data up to the time you’re forking while staying completely independent of your production service. And since Timescale allows you to change your configuration in place, you can use that same fork to try out the different configurations, or you can create a fork for each configuration you’d like to test.And the best part? This is also cost-efficient because you’ll only be charged for the time your services are running.Apart from how easy and cost-efficient they are to use, there are a few more benefits of testing on forks:You can effectively test up-to-date production data.There is no risk of taking down or harming the performance of your production database while you’re testing. Forks are independent instances with no impact on the service they are forked from.Forks serve as a better control for your tests, as your tests are isolated and running on a service separate from your production database. Your tests can then be the only item running on the machine, not impacted by production traffic.Later in this post, we will tell you step-by-step how to apply forking for testing. But now, let’s get back to our testing strategy.Designing and Writing Your Tests: Our AdviceSo, you’d like to improve your performance; you may have some hypotheses that you’d like to test; and if you’re using Timescale, now you know that you can use forks to run the tests easily. The next step is to design and write your tests. Here are some things to consider:What is the problem you are trying to solve?You might be thinking,“Well, my performance is bad; that’s why I’m reading this.”Fair enough! Let’s dig deeper, though. In what scenarios is your performance bad? And in the same vein, what is the impact of this performance degradation on your application?Isolating the problem will give you cues on how to solve it. If you are handling multiple scenarios, try picking the single scenario that’s particularly painful (i.e., the one with the most impact). Start your testing there and refine your hypothesis.What does your database look like when that problem occurs?Now that you have defined a specific problematic scenario, investigate a bit into what your database looks like when the problem arises. By this, we mean analyzing traffic patterns; looking into queries being executed at this time; maybe peeking at your scheduled jobs, and so on.The intent of doing this should be to design a testing suite that can mimic your problematic behavior as accurately as possible, so you can be sure you’re addressing it properly. If you aren’t sure where the problem lies, try to answer more general questions. For example: “Do I have a write-heavy or read-heavy workload?”One tip:pg_stat_statementsmay help you identify problems. You may be able to look back at a time when you saw a performance degradation issue, and look through some of the statements executed during that time frame. If you find any, hang onto them! They’ll be models for part of your test suite. You’ll find information on how to usepg_stat_statementshereandhere.What does your database look like typically?Once you have identified a specific problematic scenario, it’s also helpful to consider whether that’s a standard occurrence for our database or if it’s an anomaly. If it’s an anomaly, you should also consider throwing in some tests for your more general workload to ensure that you won’t be sacrificing standard performance for a situation that happens less frequently.Time to write your tests!Now that you’ve identified which tests to run, it’s time to get at it and write your tests.I have good news, and I have bad news. The good news is if you have already identified some particularly problematic scenarios, you can use those as inspiration for writing your tests. The bad news is that—well, there is no real “silver bullet” here: you’ll have to write the tests yourself. That said, some tooling in the amazing PostgreSQL ecosystem can help you with this!A common one ispgbench. Pgbench accepts user-defined scripts, which is handy as it is possible to write scripts based on your problem scenario to benchmark. Pgbench offers nice configuration options, like changing the number of clients. And for time-series workloads, there are some specific tools—including Timescale’s ownTime Series Benchmarking Suiteand AWS’sTSBench. Depending on your situation, choose a tool that works well for you.These tools provide default scripts for testing, but we strongly recommend writing custom scripts instead. The whole point of this is to test production data safely, right? Off-the-shelf scripts won’t allow you to do that. Again,pg_stat_statementsis your friend here to identify some troublesome queries!Testing in TimescaleAs we said earlier, Timescale’s forks make it highly convenient to run tests. If you’re using Timescale, these are the exact steps we recommend you take for testing as easily, cost-effectively, and safely as possible:1. Prepare your toolingFollow the advice in the previous section to design and prepare your tests.For this example, we’ll assume that you’re usingpgbench. Pgbench is available in the contrib directory of your PostgreSQL distribution. You can download it if you haven’t already by downloading the postgres-contrib subpackage using a tool like apt-get. You can see more information about the modules contained in the contrib directory in theofficial PostgreSQL docs.We highly recommend opening up the officialpgbench docsto learn about the tool!2. Create two forks of your production databaseTesting twice is good for redundancy, and creating two forks right off the bat will ensure that you have two testing databases ready with the same data loaded into them. We recommend forking your production database first (or whichever database you’re trying to improve performance on), and then “forking the fork.” If you create a second fork from the production database again, consider that the data in it will deviate from the first by whatever operations ran on the production database in the interim.Pro tip:compute and storage are decoupled in Timescale, meaning that they’re billed independently. If you don’t want to pay for compute on both forks while you are testing, you can simplypause the fork you aren’t using, so you only pay for storage during that time. You can resume this fork anytime.3. Apply the configuration you’d like to test in one of the forksWe will start by running the tests in this first fork and then follow the same steps on the second one. This will allow you to compare both results to see if they’re reproducible.Note: Throughout this, we will be using placeholders like[your-host]that you should replace with your information. If you’re using Timescale, you can find this information in the Connection Info section of the service you’re trying to test. If you’re using another service, also make sure to replace “tsdb” with the name of your database.4. Run your tests in the first forkMake sure pgbench is successfully installed on our local machine by running a version check:Pgbench -VCheck your connection. In Timescale, you can easily find these values in the Connection info pane of your service. You’ll then be prompted for your password to connect to the database.Pgbench --host=[your-host] --port=[your-port] --username=[your-username] tsdbIf you have an uninitialized database, you may get an output like this, which is expected:pgbench (14.1, server 12.12 (Ubuntu 12.12-1.pgdg22.04+1))
pgbench: fatal: could not count number of branches: ERROR:  relation ""pgbench_branches"" does not exist
LINE 1: select count(*) from pgbench_branches                         	
pgbench: Perhaps you need to do initialization (""pgbench -i"") in database ""tsdb""If you’d like to initialize your database with pgbench’s default test suite, you can add a-iflag to your arguments to initialize the database. Note that this will load data onto your database! This would look like this:Pgbench -i --host=[your-host] --port=[your-port] --username=[your-username] tsdbMake sure we are in the same directory as our testing suite, and then run our selected tests:Pgbench -f [filename] --host=[your-host] --port=[your-post] --username=[your-username] tsdbYou should get a sample output of something like this:pgbench (14.1, server 12.12 (Ubuntu 12.12-1.pgdg22.04+1))
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 1
number of threads: 1
number of transactions per client: 10
number of transactions actually processed: 10/10
latency average = 295.077 ms
initial connection time = 361.402 ms
tps = 3.388945 (without initial connection time)Rather than copy and paste this for each test you do, you can always append the results to a .txt file if needed by running something like:Pgbench -f [filename] -host=[your-host] --port=[your-post] --username=[your-username] tsdb >> results.txtKeep in mind that pgbench has a lot of options for you to adjust depending on your use case. Examples of this include--clientsto specify the number of--clientsand connect to force a new connection with each transaction rather than at the beginning and end.5. Rinse and repeat on the other fork6. Make a decisionCompare the results of the control and the test. If satisfied with the results, you can apply them to your production database.A quick reminder: if those changes imply changing your service configuration (e.g., assigning more compute to it), you can resize your service in place in Timescale using the Resources tab in the Operations pane.If your tests don’t improve performance, it’s time to test the next hypothesis!When you are done testing a fork, you can delete it. On Timescale, you are only charged for the duration of the fork. If you would like to keep it for reference but aren’t performing any active testing, you can always pause the fork, and you will only then pay for storage.ConclusionThe big challenges around testing performance improvements lie in identifying the problem scenario and writing good test cases. Once you do that, you’ll be able to easily (and safely!) try out your hypothesis using forks in Timescale. This will allow you to spend more time on what's more meaningful and particular to your use case (i.e., thinking of hypotheses and designing experiments for them) rather than wasting time manually spinning up new databases to test on.If you haven’t tried Timescale yet, you can use it completely for free for 30 days—no credit card required!Create an account here and experiment.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-test-performance-improvements-on-up-to-date-production-data-in-timescale-cloud/
2023-01-19T16:28:52.000Z,The PostgreSQL Job Scheduler You Always Wanted (Use it With Caution),"As a PostgreSQL guy, it really makes you wonder why a built-in job scheduler is not a part of the core PostgreSQL project. It is one of the most requested features in the history of ever. Yet, somehow, it just isn’t there.Essentially, a job scheduler is a process that kicks off in-database functions and procedures at specified times and runs them independently of user sessions. The benefits of having a scheduler built into the database are obvious: no dependencies, no inherent security leaks, fits in your existing high availability plan, and takes part in your data recovery plan, too.ThePostgreSQL Global Development Grouphas been debating for years about including a built-in job scheduler. Even after the addition of background processes that would support the feature (all the way back in 9.6), background job scheduling is unfortunately not a part of core PostgreSQL.So being the PostgreSQL lovers we are atTimescale,we decided to build such a schedulerso that our users and customers can benefit from a job scheduler in PostgreSQL. In TimescaleDB 2.9.1, we extended it to allow you to schedule jobs with flexible intervals andprovide you with better visibility of error logs.The flexible intervals enable you to determine whether the next run of the job occurs based on the scheduled clock time or the end of the last job run. And by “better visibility” of the job logs, we mean that they are also being logged to a table where they can be queried internally. These were extended to prevent overlapping job executions, provide predictable job timing, and provide better forensics.We extensively use the advantage of this internal scheduler for our core features, enabling us to defercompression,data retention, and refreshing of continuous aggregates to a background process (among other things).📝 Editor's note:Learn more about how TimescaleDB's hypertables enable all these features above as a PostgreSQL extension, plus other awesome things like automatic partitioning.This scheduler makes Timescale much more responsive to the caller and results in more efficient processing of these tasks. For our own benefit, the job scheduler needs to be internal to the database. It also needs to be efficient, controllable, and scale with the installation.We made all this power available to you as a PostgreSQL end user. If you're running PostgreSQL in your own hardware, you caninstall the TimescaleDB extension. If you're running in AWS,you can try our platform for free.The PostgreSQL Job Scheduler DebateBut not so fast. Before you start rejoicing, let’s review the reasons that the PostgreSQL Global Development Group chose not to include a scheduler in the database—there'll be educational for you as a word of caution.Rather than rehashing the discussion list on the subject, let's summarize the obstacles that came up in themailing list:PostgreSQL is multi-process, not multi-thread.This simple fact makes having a one-to-one relationship of processes to user-defined tasks a fairly heavy implementation issue. Under normal circumstances, PostgreSQL expects to lay a process onto a CPU (affinity), load the memory through the closest non-uniform memory access (NUMA) controller, and do some fairly heavy data processing.This works great when the expectation is that the process will be very busy the majority of the time. Schedulers do not work like that. They sit around with some cheap threads waiting to do something for the majority of the life of the thread. Just the context switching alone would make using a full-blown process very expensive.Background workers' processes are a relatively small pool by design.This has a lot to do with the previous paragraph, but also that each process allocates the prescribed memory at startup. So, these processes compete with SQL query workers for CPU and memory. And the background processes have priority over both resources since they are allocated at system startup.The next issue is more semantic.There are quite a few external schedulers available. Each one of them has a different implementation of the time management system. That is, there is a question about just how exactly the job should be invoked. Should it be invoked again if it is still running from the last time? Should the job be started again based on clock time or relative to the previous job run? From the beginning or the end of the last run?There are quite a few more questions of this nature, but you get the idea. No matter how the community answers these questions, somebody will complain that the implementation is the wrong answer because\<insert silly mathematician answer here\>.Why We Still Need a PostgreSQL Job SchedulerTimescale doesn't have the luxury of debating how many angels can dance on the head of a pin. As a database service working with large volumes of data in PostgreSQL, we face a hard requirement of background maintenance for the actions of archival, compression, and general storage. Timescale's core features, excludinghyperfunctions, depend on the job scheduler.But, rather than create a bespoke scheduler for our own purposes we built a general-purpose scheduler with a public application programming interface.This general-purpose scheduler is generally available as part of TimescaleDB. You may use it to set a schedule for anything you can express as a procedure or function. In PostgreSQL, that's a huge advantage because you have the full power of the PostgreSQL extension system at your disposal. This list includes plug-in languages, which allow you to do anything the operating system can do.Timescale assumes that the developer/administrator is a sane and reasonable person who can deal with a balance of complexity. That is longhand for ""we trust you to do the right thing.""With Great Power Comes Great ResponsibilitySo, let's talk first about a few best design practices for using the Timescale (PostgreSQL) built-in job scheduler.Keep it short.The dwell time of the background process can lead to high concurrency.  You are also using a process shared by other system tasks such as sorting, sequential scans, and other system tasks.Keep it unlocked.Try to minimize the number of exclusive locks you create while doing your process.Keep it down.The processes that you are using are shared by the system, and you are competing for resources with SQL query worker processes. Keep that in mind before you kick off hundreds or thousands of scheduled jobs.Now, assuming we are using the product fairly and judiciously, we can move on to the features and benefits of having an internal scheduler.Built-In PostgreSQL Job Scheduler: All the Nice StuffNow that we've covered the things that demand caution, here's a list of some of the benefits of using this scheduler:Physical streaming replication will also replicate the job schedule. When you go to switch over to your replica, everything will already be there.You don't need a separate high-availability plan for your scheduler. If the system is alive, so are your scheduled jobs.The jobs can report on their own success or failure to internal tables and the PostgreSQL log file.The jobs can do administrative functions like dropping tables and changing table structure by monitoring the existing needs and structures.When you install Timescale, it's already there.📝Editor's note: Quick reminder that you caninstall the TimescaleDB extensionif you're running your own PostgreSQL database, orsign up for the Timescale platform(free for 30 days).How The Job Scheduler WorksThere isa quick introductory article in the Timescale documentation. Click that link if you want more detailed information.The TL;DR version is that you make a PostgreSQL function or procedure and then call theadd_job()function to schedule it. Of course, you can remove it from the schedule using… Wait for it...delete_job().That's it. Really. All that power is at your fingertips, and all you need to know is two function signatures.Something to be aware of while you're using the scheduler is that the job may be scheduled to repeat from the end of the last run or from the scheduled clock time (in TimescaleDB 2.9.1 and beyond). This allows you to ensure that the previous job has completed (by picking from the end of the run) or that the job executes at a prescribed time (making job completion your responsibility).If you feel a bit homesick and just want to look at your adorable job, there's also:SELECT * FROM timescaledb_information.jobs;And, of course, for completeness, there's alwaysalter_job()for rescheduling, renaming, etc.Once your job has been created, it becomes the responsibility of the job scheduler to invoke it at the proper time. The job scheduler is a PostgreSQL background process. It wakes up every 10 seconds and checks to see if any job is scheduled in the near future.If such a job is queued up, it will request another background process from the PostgreSQL master process. The database system will provide one (provided there are any available). The provided process becomes responsible for the execution of your job.This basic operation has some ramifications. We have already mentioned that we need to use these background processes sparingly for resource allocation reasons. Also, there are only a few of them available. The maximum parallel count of background processes is determined bymax_worker_processes. If you need help configuring TimescaleDB background workers,check out our documentation.📝 You can also check out this blog post on tuning TimescaleDB parameters.On my system (Kubuntu 22.04.1, PostgreSQL 14.6), the default is 43. That number is just an example, as the package manager for each distribution of PostgreSQL has discretion about the initial setting. Your mileage **will** vary.Changing this parameter requires a restart, so you will need to make a judgment call about how many concurrent processes you expect to kick off. Add that to this base number and restart your system. Of course, a reasonable number has been added for you in Timescale. Remember the CPU and memory limitations while you are making this adjustment.What to Do With A PostgreSQL Job Scheduler: A Few IdeasThe original reasons for creating this scheduler involve building out-of-the-box features involving data management. That includescompression,continuous aggregates,retention policy implementation,downsampling, andbackfilling.You may want to use this for event notifications, sending an email, clustered index maintenance, partition creation, pruning, archiving, refreshing materialized views, or summarizing data somewhere to avoid the need for triggers. These are just a few of the obvious ideas that jump into my consciousness. You can literally do anything that the operating system allows.WhatNotto DoThis would be a bad place to gum up the locking tables. That is, be sure that whatever you do here is done in a concurrent manner.REFRESH INDEX CONCURRENTLYis better thanDROP/CREATE INDEX.REFRESH MATERIALIZED VIEW CONCURRENTLYis better thanREFRESH MATERIALIZED VIEW. You get it. UseCONCURRENTLY, or design concurrently. Better yet, do things in a tiny atomic way that takes little time anyway.Long-running transactions that create a lot of locks will interfere with the background writer, the planner, and the vacuum processes. If you crank up too many concurrent processes, you may also run out of memory. Please try to schedule everything to run in series. You’ll thank me later.Well Wishes to the Newly Crowned EmperorNow you have the power to do anything your little heart desires in the background of PostgreSQL without having any external dependencies. We hope you feel empowered, awed, and a little bit special. We also hope you will use your new powers for good!Try the Updated Job SchedulerThe job scheduler is available in TimescaleDB 2.9.1 and beyond. If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation. If you are using theTimescale platform, upgrades are automatic, meaning that you already have the scheduler at your fingertips.Keep LearningIf this article has inspired you to keep going with your PostgreSQL hacking,check out our collection of articles on PostgreSQL fine tuning.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/the-postgresql-job-scheduler-you-always-wanted-but-be-careful-what-you-ask-for/
2022-11-11T18:35:31.000Z,Read Before You Upgrade: Best Practices for Choosing Your PostgreSQL Version,"PostgreSQL has a long-standing reputation for having a miserable upgrade process. So, when the community heartily recommends that you should upgrade as soon as possible to the latest and greatest PostgreSQL version, it's not really surprising that your heart sinks, your mouth goes dry, and the outright dread of another laborious job takes over.It's almost like finishing a long hike or trying to convince somebody that Betamax was better than VHS. Eventually, you just want it to be over so you can take a nap. There's not even any joy about all the new features and speed. It's just too exhausting to generate emotion anymore.This blog post will hopefully serve as a guide for when to pull off the old band-aid. That is, when you should upgrade and what PostgreSQL version you should select as a target. By the end of this post, we will introduce you to our best practices for upgrading your PostgreSQL version in Timescale, so you can get over this process of upgrading as quickly and safely as possible.When to Upgrade PostgreSQL: Common MythsThePostgreSQL Global Development Grouphas simplified the upgrade process quite a bit with more explicit version numbering. Since there are only two external stimuli, there are only two choices: upgrade the binaries (minor version change) or upgrade the data on disk (major version change).The developers of PostgreSQL never really had a plan in mind for when and how to upgrade.  This seems a bit of a harsh statement when tools like pg_upgrade exist but bear with me.   These tools were meant to make upgradespossible, not to imply any particular schedule or recommendations for an upgrade plan. The actual upgrade implementation was always left as an exercise for the administrator.Let's start with some of the community's conventional wisdom and pretend that those ideas were actually a plan of sorts.Myth 1: “Upgrade as fast as possible, every time”This ""plan"" is based on the fear of existing bugs. It is a very Rumsfeldian plan that assumes you don't know what the bugs are, but you're certainly better off if they're fixed. This makes for a very aggressive upgrade pace and hopes for a better tomorrow rather than a stable today.Myth 2: ""Upgrade when you have to""The complete opposite fear-based pseudo-plan is to stick to the existing version—come hell or high water—unless you run into an otherwise unfixable bug that affects your installation. This is based on the idea that the bugs we know are better than the bugs we don't know. Unfortunately, it ignores the bugs you don't even know exist.Myth 3: “Upgrade for every minor version”This is thegeneral recommendationof the PostgreSQL Global Development Group. The general idea is that all software has bugs, and upgrading is better than not upgrading. That is a bit over-optimistic about new bugs being introduced and kind of ignores that new features that you don’t care about have to be configured—or else.This comes a bit closer to planning than guessing for minor versions, as the minor versions of PostgreSQL do not change the file system; they only change the binaries. These upgrades tend to be super heavy on bug fixes and very low on new features, which is where bugs tend to get introduced. It doesn't say anything about bugs you have actually encountered, nor does it say anything about any improvements from which you might be able to benefit.Myth 4: “Upgrade when you have time to kill”Probably the most dangerous plan since you will never have more time in the future and will probably never upgrade. Experience says that this is a completely silly plan that never gets implemented.Myth 5: “Upgrade when there are security fixes”Okay, this makes some kind of sense. Unfortunately, it ignores the rest of your installation and puts the application development team into tailspin mode for your DevOps enjoyment. It is the kind of policy you end up with when the DevOps team doesn’t really care about the Apps team.When to Upgrade PostgreSQLMuch of this guide is based on personal experience with PostgreSQL upgrades over the years. In some cases, the old was better than the new, and in others, the other way around. In some cases, the fixes worked immediately. In others, well, not so much.Very few hard and fast rules can be drawn when coming up with a plan of this nature, but I'll try to bring the experience to bear in a way that helps to make a decision in the future. That being said, this is a ""best practice"" based on experience, not a ""sure-fire thing.""As a way to reduce the amount of just sheer subjectivity and opinion around choosing the moment to upgrade, I've taken a look through the release notes of PostgreSQL. In this lookie-look, I've attempted to note where bug fixes occurred and mentally move them back to the version where they were discovered. Unfortunately, this task is also somewhat subjective, as I was not a part of the bug fix development or the bug discovery. So these are just educated guesses, but I hope rather good ones.Then I looked at the mental list that I had made and thought about whether it matched my personal experience with successful versus unsuccessful upgrades. It (again) seemed a subjectively good indicator of when an upgrade succeeded or failed.So, on to the findings.The first thing I noticed in my research is that the biggest upgrade failures were with a new major version containing updates to thewrite-ahead log (WAL). These were most notable for versions 10 and 12.Version 10 would make a book by itself. It was a major undertaking, with quite a few subsystem rewrites. In these version upgrades, there were numerous additions to items (like WAL for hash indexes), as well as improvements and changes to the background writer to support structural changes on disk. These major updates introduced the largest number of unintended behaviors, which lasted the longest before being detected and fixed.The next most striking failures came from logical replication between 10 and 11.  Of course, logical replication was invented for 10, so there had never been an attempt to use it for production upgrades before. This first use in the field was—how should I put it?—interesting.After that, the bugs died down a lot but were never quite gone.Upgrade PlanHere is my list of questions to ask before an upgrade.1. How big is the change? Was it a major refactor, and did it involve any of the following?Query planner:minor.WAL:major.Background writer:major.Memory, caching, locks, or anything else managed by the parent process:minor.Index engine: major (or just rebuild all your indexes anyway).Replication: major.Logging: minor.Vacuuming: minor.2. Were there any huge performance gains?3. Does it include major security fixes?4. Are there major built-in function() improvements/enhancements?5. Do all of my extensions exist for the new version?These are my rules of thumb for whether a new PostgreSQL version is compelling for upgrade.  Unfortunately, this still requires some subjective evaluation and a bit of professional knowledge. For instance, just because vacuum is a major feature, it doesn't mean it has ever been a problem with an upgrade. Itcouldbe, though, and we should look at its major changes with a bit of a wry mouth hold.This brings me to my personal procedure that has (so far) followed the above guidelines.Upgrade major versions when they reach the minor version .2.That is, 10.2, 11.2, 12.2, etc. This technique avoids the most egregious bugs introduced in major versions but still allows for staying reasonably close to the current.Upgrade minor versions as they are available.Minor upgrades have not created major issues thus far in my personal experience. The speed increases, bug fixes, security patches, and internationalization have been worth the minor risk.Upgrade immediately when you are more than two major versions behind. The pace of development of PostgreSQL will leave you in an unsupported version very quickly. Much quicker than ever before because they are committed to two versions a year. Only five major versions are supported, so your installation will be unsupported in approximately three years. That is a very short rope.Upgrade when the security team tells you to. It doesn't happen very often, but when it does, it's a major event.Upgrade because you need functionality. Things to upgrade for:CONCURRENTLY,SYSTEM, and performance. Things not to upgrade for: functions(), operators, and libraries.That's all there is to it.I hope this blog post has helped you to make a decision for when PostgreSQL has compelling new features for you.Of course, this is only a general rule of thumb. If you feel compelled to upgrade for some other reason, don't let my guide tell you whatnotto do. It only intends to help in the absence of any other stimuli for upgrade. You do you.I Am Ready to Upgrade. Now, What?So now you have followed the checklist above and determined that it’s time for you to upgrade your PostgreSQL version. If you’re running a production database, this may be easier said than done, especially if we are talking about upgrading your major version (e.g., from PostgreSQL 13 to PostgreSQL 14):Minor versions of PostgreSQL (e.g., from PostgreSQL 13 to PostgreSQL 13.2) are always backward compatible with the major version. That means that if you upgrade your production database, it is unlikely that anything is going to break due to the upgrade.However, major versions of PostgreSQL are not backward compatible. That means that when you upgrade the PostgreSQL version of a database behind a mission-critical application, this may introduce user-facing incompatibilities which might require code changes in your application to ensure no breakage.Practical example: if you are upgrading from PostgreSQL 13 to 14, in PostgreSQL 14, the factorial operators ! and !! are no longer supported, nor is running the factorial function on negative numbers. What may seem like a silly example is, in fact, illustrative that assumptions made about how certain functions (or even operators) work between versions may break once you update.Fortunately, PostgreSQL is awesome enough to provide clearRelease Notesstating the changes between versions. But this doesn’t solve our problem: how to upgrade production databases safely?Timescale to the RescueThis is one of the many areas in which choosing a cloud database will help. If you are self-hosting your mission-critical PostgreSQL database and want to run a major upgrade, you would have first to create a copy of your database manually, dumping your production data and restoring it in another database with the same config as your production database.Then, you would have to upgrade this database and run your testing there. This process can take a while depending on your database's size (and if we’re talking about a time-series application, it’s probably pretty big).Timescale makes the upgrading process way more approachable. Timescale is a database cloud for time-series applications built on TimescaleDB and PostgreSQL. In other words, this is PostgreSQL under the hood—with a sprinkle of TimescaleDB as the time-series secret sauce.Timescale databases (which are called “services”) run on a particular version of TimescaleDB and PostgreSQL:As a user of Timescale, you don’t have to worry about the TimescaleDB upgrades: they will be handled automatically by the platform during a maintenance window picked by you. These upgrades are backward compatible and nothing you should worry about. They require no downtime.The upgrades between minor versions of PostgreSQL are also automatically handled by the platform during your maintenance window. As we mentioned, these upgrades are also backward compatible. However, they require a service restart, which could cause a small (30 seconds to a few minutes) of downtime if you do not have a replica. We always alert users ahead of these in advance.✨Editor's Note:For security reasons, we always run the latest available minor version within a major version on PostgreSQL in Timescale. These minor updates may contain security patches, data corruption problems, and fixes to frequent bugs—as a managed service provider, we have to store our customers’ data as safely as possible.But what about upgrades between major versions of PostgreSQL? Since these are often not backward compatible, we cannot automatically upgrade your service in Timescale from, let’s say, PostgreSQL 13 to 14, which may introduce problems in your code and cause major issues!Also, upgrading between major versions of PostgreSQL can (unfortunately but unavoidably) introduce some downtime. If you are running a mission-critical application, you want complete control overwhenthat unavoidable downtime will occur. And you certainly want to test that upgrade first.A database platform like Timescale can certainly help solve this issue. Upgrading your major version of Postgres will always be a decent lift—but a hosted database platform can make this process way smoother, helping you automate what can be automated and also facilitating your testing:In Timescale, you can upgrade the PostgreSQL version that’s running on your service by simply clicking a button.You can use database forks to test your upgrade safely. Also, by clicking a button, Timescale allows you to create a database fork (a.k.a. an exact copy of your database) which you can then upgrade to estimate the required downtime to upgrade your production instance.You can also use forks to test your application changes. Once your fork is upgraded, you can run some of your production queries—you can find some of these usingpg_stat_statements—on the fork to ensure they don’t contain any breaking changes to the new major version.Let’s explore this more in the next section. If you’re not using Timescale, you can create afree account here—you’ll have free access for 30 days, no credit card required.Safely Upgrading Major PostgreSQL Versions in TimescaleHere’s how you can safely upgrade your Timescale service:First, fork your service. Timescale allows you to fork (a.k.a. copy) your databases in one click—a fast and cost-effective process. You will only be charged when your fork runs, and you can immediately delete it after your testing is complete.Now that you have a perfect copy of your production database ready for testing (with the click of a button), it’s time to click another button to tell the platform to upgrade your major PostgreSQL version automatically. You can do this in Timescale—we’ll tell you exactly how in a minute.Once the upgrade is complete in your fork, run your tests.In order to see how long the upgrade took on the fork, you can go to your metrics tab and check how long your service was unavailable (the grey zone in your CPU and RAM graphs). This will give you an estimate as to how long your primary service will be down when you choose to upgrade it.When you’re sure that nothing breaks, you can upgrade your primary service. Make sure to plan accordingly! Upgrading will cause downtime, so make sure you have accounted for that as a part of your upgrade plan.Let’s see how this looks in the console.First, check which TimescaleDB and PostgreSQL version your database is running on your service Overview page.To fork your service is as easy as going to the Operations tab and clicking on the Fork service option. This will automatically create an exact snapshot of your database.To upgrade your major version of PostgreSQL, go to your Maintenance tab. Under Service upgrades, you will see a Service upgrades button. If you click that button, your service will be updated to the next major version of Postgres (in the example below, the service would be upgraded from PostgreSQL 13.7 to PostgreSQL 14).Your Upgrade Is CompleteThat’s it! You can now use the latest and greatest that PostgreSQL has to offer. That said, choosing to upgrade is no small feat. Before going through the upgrade process, there is a lot to consider, and it is important to have a plan to account for the downtime you will experience.While the upgrade process can be a bit painful, you can at least rely on Timescale to handle the technical orchestration of the upgrade. In the future, we hope to offer even better tooling to make the upgrade process entirely pain-free (but we have to walk before we can run, right?).If you’d like to see what Timescale has to offer,start a free trial if you haven’t already. There’s no credit card required!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/read-before-you-upgrade-best-practices-for-choosing-your-postgresql-version/
2022-09-29T12:57:28.000Z,Nightmares of Time Zone Downsampling: Why I’m Excited About the New time_bucket Capabilities in TimescaleDB,"Time-series data is generally collected at a much higher rate than it will be displayed. That’s why we fall back on downsampling or rolling up the data into lower granularities (or time buckets), a technique that helps you manage the endless flows of time-series data you’re collecting. However, when resorting to downsampling data collected at high frequency, there is one important element to keep in mind: the user’s time zone.Before I joined Timescale, I had the same issue in my own startup, as most customers were located in Central Europe (Europe/Berlin, to be precise) and not in the UTC+0 time zone. I can imagine that a lot of you are struggling with this too. If your project requires exact daily overviews and you need to define what midnight means in your customer’s time zone, I know you feel this pain!Trust me, we are not alone. The support for time zones intime_bucketwas one of the most requested (and upvoted) features in theTimescaleDB GitHub repoand in polls conducted inTimescale Community Slack. But now, the days of hoop-jumping to fix this issue are over: in the last few months—and with the help of its community—Timescale re-implemented thetime_bucketfunctionalitywith full time zone support, which is now merged into the latest TimescaleDB 2.8 release.Support for months, years, and time zones was the most requested TimescaleDB feature to dateWhy is this such a big deal? Because it will save you time, no matter what time zone you or your customers are in.Time Sucks, time_bucket, and Time ZonesAs I shared earlier, while at my own startup, we had to force TimescaleDB to maketime_bucket“support” time zones. It was way too ugly, though.The gist is to force the inputtimestamptzto be converted intotimestampwithout time zone information (but adjusting it accordingly to the requested time zone). Then you need to pass it to thetime_bucketfunction and have it bucketed. The resultingtimestamptzneeds to be ripped off of its time zone information again and finally re-interpreted as a timestamp inside the requested time zone. Sounds complicated?In terms of code, it looks like this:time_bucket('86400s', TIMEZONE('Europe/Berlin', created)::timestamptz)::TIMESTAMP WITHOUT TIME ZONE AT TIME ZONE 'Europe/Berlin' AS bucketAnyway, this workaround is now a thing of the past! The new TimescaleDB 2.8 release ships with a revamped version oftime_bucket,which offers full support for time zones. The new implementation was developed with the help of the community over the last couple of months (huge shout out to all of you who contributed!) and lived in Timescale’s experimental namespace astime_bucket_ng. That means that many of you are probably familiar with it already.The change in TimescaleDB 2.8 is that we graduated the new implementation out of thetimescale_experimentalnamespace and replaced the existingtime_bucketaltogether with the implementation oftime_bucket_ng.That said, we are happy to announce that we “fixed” the most requested feature to date.New Functionality and MigrationWhile the new functionality is rather simple, its impact is immense and simplifies analytical requests requiring time zone information.As I mentioned before, most of your customers are probably not located in a UTC time zone since there aren’t a lot of countries in that time zone anyway, or customers live in one of the many countries or economic areas spread across multiple time zones. That means you have to ensure that a user’s time zone is taken into account when rolling up over day, week, month, or year boundaries.From a migration standpoint, nothing will change for users not usingtimezone. The new implementation oftime_bucketbehaves like the old version, as long as you do not provide atimezoneparameter. That means the default behavior is still bucketing on UTC boundaries, and theoriginparameter is unchanged.When provided with atimezoneparameter, though,time_bucketwill adjust theoriginto the given time zone. That means that day boundaries, as well as months and years, are adjusted too. With that change, providing a time zone is as simple as the following query:SELECT
   time_bucket('1 day', created, 'Europe/Berlin') AS bucket,
   avg(value)
FROM metrics
GROUP BY 1The result is time buckets adjusted to midnight in the Central European Time area (represented by theEurope/Berlintime zone identifier), including changes due to summer or winter (CET/CEST).When talking about the migration of existing code, all your current queries will work just as they did before the change. All parameter combinations oftime_bucketare available as implementations under the covers, and only extended versions of the method signatures are added to bring time zone support.Withtime_bucketgraduating from thetimescale_experimentalnamespace, the first major step is done. Thetime_bucket_gapfill_ngfunction is still experimental, and we will be adding time zone support totime_bucket_gapfillin the very near future.Getting Started With TimescaleDB 2.8Timescale is happy to release the new version oftime_bucketafter months of experimental status, and we want to thank everyone in the community for their help and effort in testing this feature, making it production-ready, and giving feedback!Looking back, this functionality would’ve made things much simpler and more accurate in my startup (we never handled stuff like leap seconds), but you can and should use the new functionality immediately!There is, however, more to the release of TimescaleDB 2.8 than just a newtime_bucket. Other features include experimental support for thenew policy management for continuous aggregates, and speed improvements for distributed hypertables using theCOPYprotocol forSELECTs. For more details on these changes, see What’s New in TimescaleDB 2.8 (for a complete list of all changes, see theRelease Notes.)If you are using Timescale, upgrades are automatic, and you’ll be upgraded to TimescaleDB 2.8 in your next maintenance window.New to Timescale?Start a free 30-day trial, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/nightmares-of-time-zone-downsampling-why-im-excited-about-the-new-time_bucket-capabilities-in-timescaledb/
2022-08-31T13:00:00.000Z,Monitoring Your Timescale Services With Amazon CloudWatch,"Many of our customers are using Timescale for mission-critical applications. These services require close monitoring to ensure end-users don’t suffer service interruptions due to the lack of resources. Our previous post explained how you can monitor your Timescale service metrics with Datadog. Today, we’ll do the same withAmazon CloudWatch, which is especially interesting if you are already using this service to monitor other elements of your AWS infrastructure.In this blog post, we walk you through integrating Timescale and CloudWatch to monitor your memory, CPU, and storage metrics. Timescale is the safest and most convenient way to run TimescaleDB databases in production, as it automates many of the most painful database management tasks—like backups, high availability, failover, scaling, upgrades, and much more. If you are not using Timescale yet,you can create an account here and use it entirely for free for 30 days.We also cover the steps needed to integrate Timescale and CloudWatch in the video below:✨Editor's Note:We’re also working on a Prometheus integration, which will be coming soon. Stay tuned!PrerequisitesTo follow this tutorial, you must have an AWS account with the necessary permissions to create new users and policies.You also need to have a Timescale service running. Exporting usage metrics data is a free feature on Timescale, and it is also available for trial users—you can start a free trial here.Timescale and Amazon CloudWatch: Under the HoodBefore jumping into the tutorial, let’s take a quick look at how this integration works in Timescale. This will help you make sense of the steps you’ll perform later:Timescale has an internal metrics collector service that silently watches all your Timescale services created in a particular region (e.g., us-east-1).If you want to expose the metrics for a specific service and send them to an observability tool like Amazon CloudWatch, you will first create an “exporter” in Timescale.Once you have created an exporter (linked to a specific region and a specific observability tool), you will attach the exporter to the service you want to monitor. This under-the-hood exporter is OpenTelemetry—it will give your metrics the correct format.Once your service is attached to the exporter, your service metrics will be sent automatically to CloudWatch.Architecture diagram showcasing the basic integration elements between Timescale and Amazon CloudWatchConfiguring Amazon CloudWatchThe first step to set up this integration is getting the necessary credentials for CloudWatch, namely an access key and a secret key:Head to the “Identity and Access Management (IAM)” console in AWS.Navigate to Users and click on “Add user.” On the first screen, give your user a descriptive name, and select “Access key - Programmatic access.”On the next screen, select “Attach existing policies directly.” Search for “CloudWatchFullAccess,” select the policy, and click Next.On the following screen, you can add tags to help identify this user, but this is optional.Once you create your user, write down the access and security keys.Exporting Metrics From TimescaleNow, let’s jump to the Timescale user interface.As we saw earlier, you first need to create an exporter to configure your Datadog destination in Timescale. Exporters in Timescale are associated with a particular third-party tool (e.g., CloudWatch) and have an associated AWS region. If you want to monitor different services across multiple regions (for example, one service in eu-west-1 and another in us-east-1), you’ll need to create two exporters, one per region. If the regions match, you can attach multiple services to only one exporter.To create a CloudWatch exporter in Timescale, follow these steps:Navigate to the Integrations page in the left menu.On the Integrations page, click on “Add exporter.”You’ll be taken to the configuration page for your exporter. Here, select the tool you want to integrate your service with—in this case, CloudWatch. Also, choose the Timescale region for this exporter (remember that this needs to be the same region as the service you want to export metrics from).Next, configure your CloudWatch exporter. You’ll need to define a log group name, log stream name, and namespace (these are all CloudWatch mandatory parameters). To get started, you can keep the defaults. For the CloudWatch credentials, use the ones you retrieved earlier. And for the region, select the same as your Timescale service.You can also define an IAM role (an identity you can create in your account with specific permissions) for uploading metrics. This step is optional, but having a dedicated role with only CloudWatch permissions is a recommended security practice.Once you’re done, click on “create exporter.”Your exporter is now live. The next step is connecting your service to the exporter, so telemetry data starts flowing into CloudWatch.To do that, navigate to the Operations page of your service, and click on “Integrations.” Select the exporter you previously created in the drop-down menu, and click on “Attach exporter.”Your service metrics are now accessible via Amazon CloudWatch! 🎉To check that everything works, navigate to the Metrics page in CloudWatch and see if you can access your Timescale metrics. You should see a “Timescale” custom namespace in the Query menu. The metrics coming from your Timescale service start with timescale.cloud.*.P.S.In CloudWatch, your Timescale services are identified by a service_id. You can see the service_id of a specific service by looking at the first part of the hostname. For example, your hostname may bemnyr5wrz9u.ye21f71f94.dev.metronome-cloud.com—your service_id ismnyr5wrz9u.Wrapping UpYou can now integrate Timescale with Amazon CloudWatch to monitor your service metrics.If you have a time-series application, Timescale is the most convenient way to run your database fleet. You’ll get the top performance of TimescaleDB for your time-series data—with automatic time partitioning, native compression, continuous aggregation, and SQL hyperfunctions for time-series analysis—plus all the advantages of PostgreSQL for your relational data. All this without the stress of manually managing backups, failover, or replication.You can try Timescale Cloud for free anytime; no credit card required.But there’s more: you will soon be able to use Amazon CloudWatch to monitor performance metrics in Timescale. Stay tuned and happy monitoring!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/monitoring-your-timescale-cloud-services-with-amazon-cloudwatch/
2022-11-10T17:37:28.000Z,10 Database Management Tips and Tricks From a Data Architect,"I am old. Well, maybe not in earthly years, but definitely in technology years. I remember when we would run applications on mainframe hardware, what Unix really is (or was), countless hours in extremely refrigerated data centers as part of my database management tasks, helping customers edit configuration files over the phone with “vi,” and how difficult it was to help customers run production-level software applications.In some ways, life for technologists has changed dramatically in my professional journey. As an aging technologist (and currently a data architect), I would have never envisioned the compute power, storage capacity, and network throughput available at our fingertips today.Yet, in other ways, some things always stay the same. To run production-level applications twenty-four-seven, database developers need to consider (and address) a series of requirements that are still the same today as they were 30 years ago.So, let me recap 10 of the lessons learned over the last three decades that allowed me to hone my database management skills.What is database management?Database management is the process involving the proper organization, storage, retrieval, and protection of data in a database. It is an essential aspect of information technology that ensures data integrity, accuracy, and security.In technical terms, database management involves using software known as a Database Management System (DBMS). This system provides an interface for users to interact with databases, enabling them to create, read, update, and delete data efficiently.The primary benefits of database management include standardized data access, improved data security, efficient data integration, and enhanced data analysis capabilities. Effective database management can lead to better decision-making, operational efficiency, and overall business growth.What are Database Management Systems?Database Management Systems (DBMS) serve as an interface between users and databases, playing a pivotal role in efficiently storing, managing, and retrieving data. They provide a structured environment to handle vast amounts of data, ensuring data integrity, security, and accessibility.Types of DBMSDBMSs are categorized into several types, each with its unique characteristics and use cases:Relational Database Management System (RDBMS):RDBMS is the most common type of DBMS, which organizes data into tables. Each table has rows (records) and columns (fields). The relational model's strength lies in its simplicity and the ability to manage large amounts of data efficiently. Examples include PostgreSQL, MySQL, and Oracle Database.Object-Oriented Database Management System (OODBMS):OODBMS stores data in the form of objects, similar to object-oriented programming. It allows for more complex data models and can provide significant performance improvements for certain workloads. An example of an OODBMS is MongoDB.Hierarchical Database Management System (HDBMS):In an HDBMS, data is organized in a tree-like structure, with a single root to which all other data is linked. This structure is highly efficient for certain types of queries, such as those with a known 'path'. IBM's Information Management System (IMS) is an example of an HDBMS.Network Database Management System (NDBMS):NDBMS allows each record to have multiple parent and child records, forming a web-like structure. This type is less common today but was used extensively in the past. Integrated Data Store (IDS) is an example of a network DBMS.Time-series Database Management System (TSDBMS):TSDBMS are optimized for handling time-series data, i.e., data that is indexed by time. They are often used in analytics, IoT applications, and monitoring systems.Timescale, built on top of PostgreSQL (a relational database), is an example of a TSDBMS. It combines the reliability and flexibility of PostgreSQL with powerful time-series data management capabilities, providing scalability, robustness, and performance optimization features.Often, developers in our community ask us for advice, wondering if they should invest in a managed database service or build their self-hosted deployment. Our most likely answer is, “It depends” (as with many things in the database world). But, when evaluating both options, try to consider the big picture of everything that goes behind hosting and managing a database in production.Database Management PracticesManaged databases have made the process of running applications in production considerably simple for the developer. So much so, in fact, that sometimes we tend to overlook all the work behind it. And remember, this database management workload is something you should plan for if you’re considering hosting your database in production.Let’s take a closer look at our own database platform, Timescale. Timescale is a managed cloud-native database platform that supercharges PostgreSQL for time series. The result? Fast queries at scale, cost-efficient scaling options to store higher data volumes for less money, and plenty of time-series functionality to save development time.But Timescale is much more than just an optimized time-series database: it gives customers access to an entire database environment where they can sign up and have a database running in a matter of seconds.Timescale also automatically takes care of many database operations (such as backups and failover or upgrades), and it greatly simplifies others, like creating copies of your database for testing, enabling high availability via replication, or creating read replicas for load reduction.To create such a seamless experience, managed databases juggle many elements behind the scenes, including physical facilities, compute hardware, virtualization, an operating system, and the database itself.I’ve spent plenty of time addressing those things myself throughout my career. Let’s take a journey through time and look at what it entails to provide and manage each of these building blocks effectively.Hosting Databases in Production: Building BlocksFacilitiesFor most customers, the amount of space needed to run a database is rather insignificant. It is everything else around the actual footprint that is difficult to provide for twenty-four-seven applications. In one of my previous jobs, I assisted customers in installing appliance hardware. I spent countless hours in surprisingly cold data centers assisting customers with installation services. Spending time in an actual data center highlights elements needed for proper facilities: cooling, uninterrupted power, and physical security.HardwareDealing with hardware is now a pastime for most young technologists in the software industry. Long gone are the days when we had first to figure out our facilities and then hardware. This always entailed numerous problems for companies.For example, by its very nature, computer hardware is, at best, a depreciating asset. When we used to have to procure and install our own hardware, it was essentially obsolete on the day of installation. Rarely is there a physical item on this Earth with such a short shelf life! I remember the days spent purchasing, physically installing, running cables, and manually updatingBIOSes… To replace it soon thereafter.Today, we can use a cloud service (AWS, for example) to allocate compute resources dynamically. This has greatly simplified the task of deploying the hardware needed to run applications, making things smoother for developers and companies. In Timescale, we’re actually running Timescale in AWS ourselves, taking advantage of the flexibility of this cloud ecosystem to build our platform.What we learned from our own experience is that having access to a cloud service is just the tip of the iceberg when dealing with hardware. One can hastily miscalculate the effort that goes into it. Sure, it is pretty straightforward to instantiate anAmazon EC2 instance. But do you have the time and flexibility to provide an easy front end to instantiate and change resources on the fly? Can you monitor those resources for consistent uptime and performance? And ultimately, can you automatically adjust when the resources are not performing as one expects twenty-four by seven? If reading this paragraph didn’t exhaust you, maybe you can.Deploying your hardware in the cloud doesn’t necessarily simplify many critical database management tasks you will need to perform daily if you’re running a database in production. This aspect often confuses our users, who expect these tasks to be less time-consuming, especially if they’re used to working with managed database services.So while a cloud service costs a bit, it is nothing compared to the costs if a customer were to build that functionality for themselves. I am happy to no longer deal with the entire facilities and hardware pieces of the puzzle.Virtual ManagementSpeaking of operating a cloud infrastructure, let’s talk about virtualization.Virtualization was introduced about midway through my career in technology. I remember quite well how it made certain tasks much easier. For example, I could virtualize the hardware in a multi-node software application, so I didn’t need a full hardware rack but rather a much smaller footprint. It saved physical space and time.But it came at a cost since nothing is free. There is now a layer of complexity that, if not properly managed, can greatly impact all the layers above, including how the operating systems (O/S) and applications run. Too often, I’ve seen virtualization misused (e.g., purposely overprovisioned), negatively affecting everything else that runs above the virtualization layer.However, virtualization has come a long way in a few short years to where it now allows us to run entire data centers as virtualization centers.O/SOperating systems are funny things. They are not seen yet are crucial to our lives. Even our phones now have an operating system. I’ve seen many different “server” operating systems in my life, from IBM OS/360 to VAX/VMS to just about every Unix variant (SunOS, Solaris, System V), and now Linux and its many distributions. Not many people think about operating systems except when things go bad.But operating systems are another thing that needs care and feeding, so they need to be on someone’s mind. Benjamin Franklin’s quote that the only thing certain in life is death and taxes needs to include software and security patches. If you’re thinking about self-hosting your own database, this is another aspect to consider: make sure to remember that you’ll need to monitor and maintain your O/S, running upgrades every time it’s necessary to patch a security vulnerability or an important bug.DatabaseWe now come to the last component—the database itself. Timescale is an extension of Postgres. Just like an O/S, the database needs care and feeding. Timescale takes care of database maintenance like every other layer before, including bug and security patches.But database management in production involves another crucial element: preparing for when things go wrong. That is the only certainty of the production process—unfortunately, thingswillgo wrong sometimes, and you need a plan.If you’re self-hosting your database, you must define a set of operative rules to determine what to do when your storage corrupts, or your compute fails. At a minimum, you must take backups regularly, testing them to ensure your data is safe.But take into account that recovery from a backup can be a rather slow process. You may want to consider setting up an alternative method (e.g., replication) that avoids the potential downtime due to a database failure. Designing for less downtime in production is often a vital element to consider as, more often than not, having your database down will cause the whole business to collapse. Besides the money loss, this is a terrible experience for your end-users.Self-Hosting or Managed?Reminiscing on the past helps to understand all of the behind-the-scenes elements that are at play when managing a production database.By choosing to run on a managed platform like Timescale, you don’t have to worry about the cost and hassles of managing facilities and hardware provisioning.You can deploy databases in one click and scale them as you need. Once you don’t need them anymore, you can delete them and not think about the underlying hardware that powered them ever again.Timescale makes the most of virtualization technology to provide properly allocated resources to end-users. Timescale customers don’t need to worry about instances since Timescale does not overprovision compute resources. In addition, the service carefully monitors every virtual machine to ensure each customer has the requested resources.Timescale takes care of maintaining and monitoring the O/S.Timescale is constantly monitoring to ensure the O/S is running correctly, not to mention the never-ending bug and security patches. As an administrator, I’d be happy never to maintain a server O/S again, which is exactly what Timescale provides.Talking about things I’d be happy never to do again: worrying about backups.Timescale automatically keeps up-to-date backupsby performing full backups weekly and partial backups daily. We stressed about backups because we weren’t good at consistently performing them, let alone testing them to ensure they were ready if they were ever to be used.Besides backups, Timescale keeps Postgreswrite-ahead log (WAL) filesof any changes made to the database. This ensures the recovery of a database at (and to) any point in time without experiencing data loss.Finally, another huge advantage of managed databases (like Timescale) is thatyou can easily enable database replication for high availability.This means that when things go south and the production database goes down, another instance automatically spins up in seconds instead of leaving your users down for what could turn into hours. Nobody wants to be called to solve a complicated mess in the middle of the night.But that’s not even it. The dynamic cloud-native architecture of Timescale allows us to provide our customers with many more services that simplify your daily tasks, so you can focus on your applications instead of operating your database.For example,Timescale offers a single-button “fork” mechanism.This single-button action duplicates an entire environment at the push of a button (hardware resources, O/S, database, application code, and data).Not so long ago, we had to duplicate the entire facilities-to-application stack manually. Even if we had the components readily available, it still took hours, if not days. Duplicating an entire environment with the click of a button is precisely what application engineers need to focus on critical tasks. They should spend their time writing and testing applications, not on the platform- or system-level tasks involved in creating and testing their applications.And since it is so simple to fork the database, you can also delete that instance—so you will only pay for the resources needed for that task. In other words, one can go through an entire quality assurance cycle and spend only pennies.Now, onto to my 10 tips.10 Database Management Tips and Tricks1. Normalize Your Data to Optimize PerformanceAs someone who has managed a lot of databases, I cannot overstate the importance of data normalization. This process is absolutely crucial for maintaining an efficient, reliable, and well-structured database. Data normalization is a methodology that involves organizing data in a database to reduce redundancy and improve data integrity.By distributing data across different tables and establishing relationships between these tables, normalization ensures that each piece of data is stored only once. This reduces data redundancy, saving on storage space and increasing the efficiency of data retrieval operations.Moreover, normalization enhances data integrity by enforcing consistency. When data is stored redundantly, there's a risk of inconsistency—if data is updated in one place but not another, it can lead to discrepancies. Normalization mitigates this risk by ensuring that each unique data point is stored only once, thereby promoting data accuracy and consistency.2. Implement Regular Backups and Test ThemIt's crucial to underline that taking regular database backups is not just a good practice but a necessity. But backing up your data is only half the battle; the other half is making sure those backups actually work when you need them. This is where backup testing comes into play.The importance of testing your backups periodically cannot be overstated. Without verification, you may be under the false impression that your data is safe when, in fact, you could be facing potential data loss or corruption scenarios.Backup testing ensures that the data recovery process will function correctly in case of a system failure or data loss incident. It identifies potential issues such as data corruption, incomplete backups, or problems with the backup software itself, allowing you to rectify these before disaster strikes.The benefits of this approach are manifold. Not only does it provide peace of mind, but it also ensures business continuity, minimizes downtime, and protects against potential financial losses or reputational damage stemming from data loss.With Timescale, you can save time with automatic backups, upgrades, and failover.3. Monitor Database Performance ActivelyContinuous monitoring is a fundamental aspect of maintaining an efficient and reliable database system. It allows us to proactively identify potential issues, optimize performance, and ensure the smooth operation of your database.There are several tools and methods we can utilize to continuously monitor the performance of a database:Performance metrics:These are key indicators of the health and efficiency of your database. They include things like query execution time, CPU usage, memory usage, and disk I/O. By regularly checking these metrics, we can identify any anomalies that may indicate a problem.Query analysis:By examining the queries that are run against our database, we can identify inefficient operations that may be slowing down performance. Tools like SQL Profiler orpg_stat_statements(for PostgreSQL/Timescale) can provide valuable insights into query performance.Database logs:Logs can provide a wealth of information about what's happening within our database. By analyzing these logs, we can identify errors, track changes, and gain insight into the overall operation of our system.Automated monitoring tools:Tools such as Nagios,Zabbix, orDatadogcan automatically track performance metrics and alert us to potential issues. These tools can be set up to monitor TimescaleDB just as effectively as they do for traditional RDBMSs.Load testing:This method simulates high-load scenarios to see how the database responds. It can help identify bottlenecks and areas that need optimization.See how you can test performance improvements on up-to-date production data with Timescale.4. Secure Your Database Against Unauthorized AccessIn today's digital age, where data breaches are all too common, implementing robust security measures is not just a good practice—it's an absolute necessity.Firstly, user roles and permissions play a critical role in database security. By assigning specific roles and permissions to each user, you can control who has access to what data and what actions they can perform. This principle of least privilege—providing only the necessary access needed for a user to perform their tasks—helps minimize the potential damage from accidental or malicious actions.Timescale has a robust role-based access control. This allows for finely tuned access controls, ensuring that users can only access and manipulate the data they need to perform their job. Plus, we are SOC2 Type II and GDPR compliant.Secondly, it's crucial to stay up-to-date with security patches. Software providers release these patches to fix known vulnerabilities that malicious actors could exploit.As a user of Timescale, you don’t have to worry about Timescale upgrades or patches: they will be handled automatically by the platform during a maintenance window picked by you. These upgrades are backward compatible and nothing you should worry about. They require no downtime.5. Plan for Scalability From the OutsetPlanning for future growth requires considering factors like increased user load, larger datasets, and integration with other systems. Here's how you can do it:Scalability: Scalability is the ability of a system to handle increased load without impacting performance. For databases, this means being able to handle more queries, larger datasets, and more users.Data Partitioning: Partitioning splits your data into smaller, more manageable pieces, allowing for improved query performance and easier maintenance.Indexing: Proper indexing can significantly improve query performance, especially as your database grows.Integration: As your system grows, it may need to integrate with other systems. Designing your database with this in mind from the start can save you a lot of headaches down the line.Dataarchivingandretention: Over time, your database will accumulate large amounts of data. Implementing data archiving strategies and setting up retention policies will help manage this data growth.Monitoringandperformancetuning: Regular monitoring and tuning of your database can help identify potential bottlenecks and optimize performance.6. Optimize Queries for EfficiencyInefficient queries can significantly impact a database's performance, increasing I/O load, CPU usage, and blocking. These poorly optimized queries can be the root cause of many web application bottlenecks, often performing a large number of reads but returning a relatively small number of rows. This inefficiency can lead to slower response times and diminished user experience.To improve the performance of your database, consider the following strategies:Optimizeyourqueries:In many cases, database performance issues are caused by inefficient SQL queries. Avoid usingSELECT *statements as they can slow down your system by retrieving unnecessary data. Instead, specify the exact columns you need. Also, avoid nested queries and use joins where possible for more efficient execution.Addmissingindexes: Table indexes in databases help retrieve information faster and more efficiently. Without proper indexing, the database server must scan the entire table to retrieve the desired data, which can be very time-consuming with large datasets.Useappropriatedatatypes: Using improper data types can lead to unnecessary space usage and slower query performance.Always choose the most appropriate data type for your data to ensure efficiency.Regularmaintenance: Regular database maintenance activities like updating statistics, rebuilding indexes, and removing old data can help maintain optimal performance.Monitor Query Performance: Keep an eye on your query performance over time.Tools likepg_stat_statementscan provide valuable insights into query performance and help identify inefficient queries that need optimization7.7. Regularly Update and Patch Your DBMSAs I mentioned, ensuring your DBMS is consistently updated is a crucial part of maintaining a robust, secure, and efficient database environment.Bugfixes: Updates often include fixes for bugs identified in previous versions of the software. These bugs can range from minor usability issues to significant problems that can impact the system's performance or functionality. By updating your DBMS regularly, you ensure that these issues are resolved promptly, allowing your database to operate smoothly and efficiently.Securitypatches: Security is a paramount concern in any database system. Updates often include security patches that address vulnerabilities identified in the software. These patches are essential for protecting your data from potential threats and breaches. Regular updates help fortify your database's security, safeguarding your valuable data.Newfeatures: Each update potentially brings new features and enhancements to the existing functionality. These improvements can provide more efficient ways to manage and manipulate your data, improve query performance, or introduce new capabilities that can enhance your database's overall utility.Performanceimprovements: Updates often include optimizations to improve the DBMS's performance. These could be enhancements to the query processing engine, better memory management, or more efficient algorithms for handling data. By keeping your DBMS updated, you can benefit from these performance enhancements, ensuring your database operates at its optimal speed and efficiency.8. Maintain Comprehensive DocumentationCustomers often praise our Docs, and we take great pride in them.Having detailed documentation for your database schema, stored procedures, triggers, and other elements is crucial. This practice is not just a good-to-have—it's a necessity for robust, efficient, and maintainable database management.9. Employ Data Archiving StrategiesArchiving older data is an effective strategy to improve the performance of your database management system. It involves moving infrequently accessed or non-essential data from the primary database to a secondary storage system, which can lead to several performance benefits.To effectively archive data without loss, consider the following strategies:Definearchivingpolicies:Establish clear policies on what data should be archived, when, and how.This could be based on the age of the data, its relevance to current operations, or legal requirements.Usedatapartitioning: In Timescale,you can use automatic partitioning management features to segregate older data easily.Ensuredataaccessibility: Even though data is archived, it may still need to be accessed occasionally. Ensure your archiving solution allows for easy data retrieval when necessary (like Timescale with S3!).Datavalidation: Before and after archiving data, validate it to ensure no data has been lost during the process. This can be done by comparing record counts or using data validation tools.Regularlyreviewarchivingstrategy: As business needs change, so too should your archiving strategy. Regularly review and update your strategy to ensure it continues to meet your organization's needs.Timescale allows you to tier data to Amazon S3: By running a simple command on your hypertable (add_tiering_policy), you can automatically tier older data to a low-cost, infinite storage layer built on Amazon S3. Yet the data still remains fully queryable from within your database, and this tiering is transparent to your application.10. Leverage Database Caching MechanismsCaching is a powerful strategy that can significantly enhance the speed and efficiency of your database operations.Understanding cachingIn essence, caching involves storing frequently accessed data in a high-speed access area—known as a cache—for quick retrieval. When a database query is executed, the system first checks the cache. If the requested data is found (a cache hit), it's returned immediately, bypassing the need for time-consuming disk reads.The impact of cachingThe benefits of caching are quite substantial:Fasterdataretrieval: Cache memory is faster than disk storage. Hence, retrieving data from the cache significantly reduces response times, speeding up database operations.Reduceddisk I/O: By serving data from the cache, you minimize disk I/O operations, reducing wear on your storage devices and freeing them up for other tasks.Improvedsystemperformance: The cumulative effect of faster data retrieval and reduced disk I/O leads to the improved overall performance of your database system.Implementing effective caching strategiesTo leverage caching effectively, consider the following strategies:Identifyfrequentlyaccesseddata: Analyze your workloads to identify data that are accessed often. This could be data used in popular queries or reports. Prioritize this data for caching.Useappropriatecachesize: The cache size should be large enough to store frequently accessed data but not so large that it consumes excessive system resources.In Timescale, you can configure shared_buffersto determine the amount of memory dedicated to the cache.Monitor andadjust: Regularly monitor your cache's hit ratio (the percentage of total requests served by the cache). A low hit ratio may indicate that your cache size is too small or that the wrong data is being cached. Adjust your caching strategy based on these insights.The Future Is Cloud-FirstI guess I am old in a way. But going through these experiences helped me understand the great value of Timescale. When I work with customers, I no longer need to freeze in a data center, helping a customer install yet another rack-mounted piece of hardware. Timescale customers no longer need to buy hardware that essentially becomes obsolete the day it is bought and installed.Long gone are the days of waiting for new hardware to increase the capacity of a system since virtualization allows for dynamic resource allocation. And can I say how much I don’t miss keeping up on O/S and security patches?Lastly, what I hope Timescale customers will appreciate, just as much as everything else, isthe expertise available to help them from a worldwide Support team. I started my career in technology as a support engineer. This was well before the modern support mentality—we actually spoke with customers either on the phone or via personal email interaction.Timescale Support is fully staffed, offering the same high-touch support. They’re ready to help on a myriad of topics, such as data migration, schema design, data modeling, query or ingest performance, compression settings, and more, providing in-depth consultative support at no additional charge.We often forget all that goes into providing a twenty-four-seven database platform. My hope is Timescale customers occasionally take a moment to think about our technology journey and appreciate that they can concentrate on their applications and don’t need to worry about how Timescale has learned from the past to provide a database platform for the future.Keep reading to learnhow to save money and improve performance by carefully managing your time-series data.And, if you are starting to understand the appeal of managed databases, embark on this journey and start prioritizing your applications:sign up for Timescale. It is free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/database-management-behind-the-scenes-lessons-from-a-data-architect/
2022-11-03T14:00:00.000Z,How to Spin Up an Instance in 30 Seconds and Add Time-Series Data to Your Database,"Tracking and analyzing change has become one of the most valuable commodities in a constantly evolving world. Regardless of their size, businesses, governments, and other organizations are collecting data to measure, analyze, and ultimately understand what is changing, how it’s changing, and why it’s changing. And for that, few types of data give us better insight thantime-series data.Time-series data allows you to track change over time, from milliseconds to even years. While deeply informative, it is also unforgiving and pervasive.Those working with time-series datawill most likely end up with massive amounts of information to manage, query, and store.That’s where Timescale comes in—an easy, innovative, and cost-effective solution to store and analyze your time-series data.We offer a 30-day free trial for new sign-ups, no credit card required.Okay, okay, we know that seeing is believing. Follow us.Spin Up Your Time-Series Data Instance in 30 SecondsIf you want to check how you can really (really!) spin up your first Timescale instance in 30 seconds, watch the following video.And, if you’re working with time-series data—which we kind ofknowyou are—keep scrolling down. We will walk you through the process of ingesting time-series data into your Timescale from a CSV file.Try Timescale TodayNow that you’ve seen it in action and we helped you get started—go for it!Try Timescale for freeand start querying, analyzing, and storing your time-series data like never before.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-spin-up-an-instance-in-30-seconds-and-add-time-series-data-to-your-database/
2023-02-24T15:17:18.000Z,Do More on AWS With Timescale: Build an Application Using Lambda Functions in Python,"AWS Lambda is a serverless compute service that allows you to run code in the cloud with ease. A common misconception is that the name “serverless” implies no servers are involved, which couldn’t be further from the truth.When you deploy your application (in this video, we build a time-series application with two Lambda functions in Python), there are, in fact, servers running your code—you just don’t need to worry about it like you would if you deployed your application on EC2. This is where AWS shines: it abstracts the infrastructure, operating system, logging, and much more away, freeing up time for you to work on what actually matters: your application code.Another great benefit of this abstraction is scaling. Usually, scaling an application involves a framework like Kubernetes, which is complex and expensive to run. AWS Lambda, on the other hand, can run multiple concurrent instances of your code out of the box. This might require some clever architecting with locks for specific transactional workloads but shouldn’t be necessary in most cases.Those properties of AWS Lambda make it a perfect candidate for time-series applications which tend to process high volumes of data. Whether you deploy your Lambda function behind an API Gateway or as a Kinesis data stream consumer, it will seamlessly scale up and down depending on the quantity of data flowing in. Lambda can even scale to zero, meaning that if no data is being received, you are not paying for expensive servers running idle.This is not the first time I’ve experimented with pairing AWS and Timescale. Recently, I posted a video onhow to set up VPC peering between your own AWS VPC and your Timescale VPCso you can make your valuable time-series data less vulnerable to exploits and attacks.I also explored the possibilities of popular AWS services and Timescale a few months ago when I co-wrote a blog post about using them tobuild simpler and faster time-series applications.This time, I created a serverless time-series application comprising two Lambda functions written in Python. One function parses a post request and inserts the request parameters into TimescaleDB on Timescale. The other retrieves the latest five entries in our database. In the video, I use the AWS SAM CLI to provision the AWS infrastructure that powers our application.Check it out! I hope you enjoy it.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/do-more-with-aws-and-timescale-cloud-build-an-application-using-lambda-functions-in-python/
2022-10-19T13:00:00.000Z,"Database Indexes in PostgreSQL and Timescale: Your Questions, Answered","Database indexes are a big topic in the PostgreSQL landscape. Timescale provides impressive time-series optimizations on top of PostgreSQL: you get performance at scale, cost-efficiency, and all the time-series features you need without toiling through yet another learning curve. Timescaleis justPostgreSQL—with time-series superpowers.💡Editor's Note:Want to learn thebasics about database indexes in PostgreSQL? Check out this article.Still, we often get questions from customers on how indexes work with Timescale. The short answer to those questions is:database indexes in Timescale work the same as in regular PostgreSQL.But perhaps, you want to dig a bit deeper.In this blog post, we’ll walk you through the most common questions we get on this topic:Does a normal PostgreSQLINDEXwork with Timescale?Will creating a Timescale hypertable use the existing PostgreSQL indexes?After creating the Timescale hypertable, do I use the standardCREATE INDEXmethod for creating new indexes?Does the PostgreSQL Planner use the indexes with Timescale hypertables?Do Timescale hypertables use composite indexes?You already know the answer to all of the above is “yes,” but let’s jump into more detailed answers.What Happens to My Database Indexes When I Convert a PostgreSQL Table Into a Hypertable?In Timescale, one must first create or use a regular PostgreSQL table to convert it into a hypertable.What happens to the database indexes during that process?The following is a straightforward PostgreSQL table definition we will use in our examples, including a definition of an index onid:CREATE TABLE public.my_table (
	""time"" timestamp without time zone NOT NULL,
	id integer,
	value real
);
CREATE INDEX my_table_id_index ON public.my_table USING btree (id);At this point, we have a normal PostgreSQL table andINDEXdefined. It will look like this when viewed from psql:index_example=> \d my_table
                    	Table ""public.my_table""
 Column |        	Type          | Collation | Nullable | Default
--------+-----------------------------+-----------+----------+---------
 time   | timestamp without time zone |       	| not null |
 id     | integer                  	|       	|      	|
 value  | real                    	|       	|      	|
Indexes:
	""my_table_id_index"" btree (id)The Timescale UI also shows us this information. If we navigate to the Explorer in Timescale, we’ll see this table showing as a normal PostgreSQL table (PostgreSQL tables are tagged with a blue indicator):We can highlight the “Indexes” tab to see the PostgreSQL index:Now, let’s convert the PostgreSQL table into a hypertable:SELECT create_hypertable( 'my_table', 'time', chunk_time_interval => INTERVAL '1 hour', migrate_data => TRUE );In psql, this looks a bit different now:index_example=> \d my_table
                    	Table ""public.my_table""
 Column |             Type            | Collation | Nullable | Default
--------+-----------------------------+-----------+----------+---------
 time   | timestamp without time zone |           | not null |
 id     | integer                     |           |          |
 value  | real                        |           |          |
Indexes:
	""my_table_id_index"" btree (id)
	""my_table_time_idx"" btree (""time"" DESC)
Triggers:
	ts_insert_blocker BEFORE INSERT ON my_table FOR EACH ROW EXECUTE FUNCTION _timescaledb_internal.insert_blocker()And in the Timescale UI:The table is now converted to a hypertable, and we can also see the number of chunks (the internal units of partitioning in Timescale’s hypertables).How does our original index look now?The index onidis the same. This is because Timescale is builtonPostgres, not beside, instead of, parallel to, etc. Timescale fully supports generic PostgreSQL tables andINDEXdefinitions as they are (see Editor's Note below).  You may have also noticed that there is an additionalINDEX: the creation of the hypertable will automatically create anINDEXfor the timestamp column used in the time-series data.So long story short, the direct answer to our original question, “What happens to a PostgreSQLINDEXwhen a table is converted to a hypertable?” is… Not much! You’ll be able to access your hypertable the same way you were accessing your original PostgreSQL table, plus you will also have an additional index for the timestamp column.✨Editor’s Note:While indexes are fully supported, there is a limitation on how they are implemented with Timescale hypertables. They do not support CREATE INDEX CONCURRENTLY. This, however, should not affect how you work with indexes in Timescale. As we said, it’s business as usual!What Is Going on Under the Hood?A hypertable is an abstraction layer—to the end user, the hypertable looks like a normal PostgreSQL table. In fact, at Timescale, we want end users to view them as such so they do not need to learn a new data structure or language.While the end user interacts with the hypertable as a normal PostgreSQL table, the Timescale software is an abstraction layer that buffers the end user for the details. Essentially, a hypertable consists of smaller tables called chunks. These chunks are actual PostgreSQL tables themselves. However, Timescale manages all chunks as one logical table.Timescale chunks are separated by a timeINTERVAL. Each chunk contains a subset of the total data—again, based on a timeINTERVAL. Since each chunk is a separate PostgreSQL table, each chunk will have its own set of indexes.To view individual chunk table definitions, use theshow_chunk()function:index_example=> SELECT show_chunks( 'my_table' );
               show_chunks
-----------------------------------------
 _timescaledb_internal._hyper_1_1_chunk
 _timescaledb_internal._hyper_1_2_chunk
 _timescaledb_internal._hyper_1_3_chunk
 _timescaledb_internal._hyper_1_4_chunk
 _timescaledb_internal._hyper_1_5_chunk
 _timescaledb_internal._hyper_1_6_chunk
 _timescaledb_internal._hyper_1_7_chunk
 _timescaledb_internal._hyper_1_8_chunk
 _timescaledb_internal._hyper_1_9_chunk
 _timescaledb_internal._hyper_1_10_chunkSince each chunk is a PostgreSQL table, we can view a chunk and see its indexes:INDEXindex_example=> \d _timescaledb_internal._hyper_1_1_chunk
            Table ""_timescaledb_internal._hyper_1_1_chunk""
 Column |            Type             | Collation | Nullable | Default
--------+-----------------------------+-----------+----------+---------
 time   | timestamp without time zone |           | not null |
 id     | integer                     |           |          |
 value  | real                        |           |          |
Indexes:
    ""_hyper_1_1_chunk_my_table_id_index"" btree (id)
    ""_hyper_1_1_chunk_my_table_time_idx"" btree (""time"" DESC)
Check constraints:
    ""constraint_1"" CHECK (""time"" >= '2022-04-01 00:00:00'::timestamp without time zone AND ""time"" < '2022-04-02 00:00:00'::timestamp without time zone)
Inherits: my_tableNotice that the two indexes were defined for the chunk:index_example=> SELECT indexdef
                FROM pg_indexes
                WHERE indexname LIKE '_hyper_1_1_chunk%';
                               indexdef
---------------------------------------------------------------------------
 CREATE INDEX _hyper_1_1_chunk_my_table_id_index ON _timescaledb_internal._hyper_1_1_chunk USING btree (id)
 CREATE INDEX _hyper_1_1_chunk_my_table_time_idx ON _timescaledb_internal._hyper_1_1_chunk USING btree (""time"" DESC)How Do Indexes and Hypertables Work?As you may now guess from the previous section, anyINDEXwill be used with a hypertable, just as with any normal PostgreSQL table. All the query planning regarding the internal partitioning into chunks happens automatically.If we perform a simple query, we will see theINDEXbeing used in the query planner just like it would be with the normal PostgreSQL table:index_example=> EXPLAIN SELECT AVG(value)
                FROM my_table
                WHERE id = 5 AND
                      time BETWEEN '2022-04-01 00:00:00' and '2022-04-01 23:59:59';
                               QUERY PLAN
-------------------------------------------------------------------------------
 Aggregate  (cost=28.35..28.36 rows=1 width=8)
   ->  Index Scan using _hyper_1_1_chunk_my_table_id_index on _hyper_1_1_chunk  (cost=0.29..28.29 rows=24 width=4)
         Index Cond: (id = 5)
         Filter: ((""time"" >= '2022-04-01 00:00:00'::timestamp without time zone) AND (""time"" <= '2022-04-01 23:59:59'::timestamp without time zone))We need to note a couple of things about the query plan:Timescale inherently knows which chunks to query based on the timeINTERVALwhen the hypertable was created.The user-addedINDEX(on theidcolumn) is consulted to find the appropriate rows.Timescale hides the complexity of indexing while at the same time providing the same functionality as normal PostgreSQL indexes.How Do I Define anINDEXDirectly in a Hypertable?If you want to define indexes directly in your hypertable (after having transformed your original PostgreSQL table into a hypertable), the process is exactly the same as normal PostgreSQL. UseCREATE INDEX!If, for some reason, theidINDEXwas not created with the original PostgreSQL table, one just needs to use the same syntax with the hypertable:CREATE INDEX my_table_id_index ON my_table(id);Do I Need to Do Anything Differently With Indexes and Hypertables vs. Regular PostgreSQL?No! And that is the point: use Timescale and hypertables just as one would with PostgreSQL tables!What About Creating Indexes on Existing Large Databases?Actually, Timescale provides a “lighter” load than normal PostgreSQL when creating a newINDEXon an existing large table. One can use an option when creating a newINDEXthat will create indexes on a per-chunk basis, which means only locking that specific chunk! This means the entire table is not locked during aCREATE INDEX.More information onCREATE INDEXon a hypertable can be foundin our docs.✨Editor’s Note:If you want to learn more about indexes in Timescale,check out our documentationand learn how to fine-tune your PostgreSQL database performance byoptimizing your database indexes.Next StepsWe hope this article gave you the answers you were looking for on how database indexes in PostgreSQL and Timescale work together.Timescale expands PostgreSQL for time-series data, adding the superpowers you need for your time-series applications: high performance at scale, cost-efficiency, and time-saving features specifically designed for time series. All without ever leaving behind the PostgreSQL you already know and love. So if you’re handling time-series data, give it a try!Sign up for a free 30-day trial (no credit card required), and start indexing!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/database-indexes-in-postgresql-and-timescale-cloud-your-questions-answered/
2022-10-13T13:00:00.000Z,VPC Peering: From Zero to Hero,"Virtual Private Cloud (or VPC) peering enables you to securely access data stored in your cloud database from your existing cloud infrastructure without ever exposing your services to the public internet. This can be incredibly useful if your use case demands maximum security and privacy—VPC peering allows you to replicate (as much as possible) the isolation of a self-hosted deployment.In this blog post, we walk you through everything you need to know to configure VPC peering between Timescale and AWS, moving from the fundamentals all the way toward advanced use cases. This post aims to help users with little or no networking background get up and running quickly with VPC peering—but even more experienced users may find helpful information here!✨Editor's Note:This blog post is focused on VPC peering in AWS,which is what’s currently supported in Timescale.If you’d like to see support for other cloud providers,let us know—your feedback really helps us build the right features for you! 🙏 And if you’d like to get a glimpse of how the Timescale Support Team works,check out this blog post.What Is VPC Peering?As we explained, VPC peering is like a superhero invisibility cloak, allowing you to access your cloud data without the risk of having your services exposed to the public internet.More specifically, this feature enables you to create a private network “peering” connection between your Amazon VPC(s) and your Timescale VPC(s), making it possible for the machines in the two VPCs to speak to each other directly without going through the wider Internet. The services within your Timescale VPC will only be accessible from your Amazon VPC. By isolating services in such a manner, you gain greater security and control over your database.Therefore, VPC peering involves configuring two networks: an external cloud providers network (AWS) and the Timescale network. So let’s start by covering the basic networking concepts you need to be familiar with.Networking FundamentalsIn the Timescale Support Team, we often receive support cases from users seeking assistance with VPC configuration. Often, we find that a lack of knowledge about networking fundamentals is at the heart of the issue—let’s fix that by explaining the fundamental components of a network and how they relate to VPCs.If you’re familiar with these topics, please feel free to jump to theConfiguring Your Networks for VPC Peering section.IP addressesAn IP address is your Internet Protocol Address. In much the same way as your postal address allows other people to find your home, your IP address allows other computers to find your computer on the internet.An IPv4 address is written in a human-readable format of dotted decimal (base 10) notation e.g., 192.168.0.1. This represents four groups of decimal values separated by a decimal point. Computers interpret this IPv4 address as a series of ones and zeros (binary), and, therefore, this IPv4 address in binary format is represented by four segments of ones and zeros separated by a decimal point (11000000.10101000.00000000.00000001).SubnettingEvery IP address has two portions that represent a network portion and a host portion. The network ID helps route traffic to your host ID, in the same way as the country and city on your postal address help route packages to your home.10 Main Street could be anywhere, but 10 Main Street in City (Nairobi), in Country (Kenya), can be found efficiently and securely.✨Editor's Note: “Securely?” you may be wondering. To be 100 % accurate: an IP address would only point to your location, but not necessarily in a secure way. For the sake of simplicity, we're assuming that networks already incorporate security protocols (such as passwords and certificates). This is like assuming the door is locked once you get to the address. 🙏A subnet mask differentiates the network portion and the host portion of your IP address. Without going into too much detail, the subnet mask is a combination of ones and zeros, which, when applied to an IP address, will distinguish the host and network portions. The network bits will be ones, and the host bits will be zeros. Below is an illustration of these two IPv4 address portions together with a subnet mask.Network IDHost IDIPv4 address19216801Subnet mask255255255024 bitsThis is a 24-bit subnet mask because, in binary representation, we have eight ones in the first three octets (i.e., 11111111.11111111.11111111.00000000). In Classless Inter-Domain Routing format (or supernetting), this will be presented with a /24 as 192.168.0.0/24.CIDR (Classless Inter-Domain Routing)CIDR breaks away from the standard network classes to expand the standard subnets to accommodate a wider customizable range of network IPv4 addresses. For example, if we modify the /24 subnet mask to /20 subnet mask, this will increase the number of host network IDs from 254 (2^8 -2) to 4,094 (2^12 -2).In our analogy, if a subnet is seen as a city, we’ll describe a CIDR block as a further way to segment this subnet, so a CIDR block could be seen as a postcode or neighborhood.AWS VPC routingRouting is the selection of the best route from one computer (IP Address) to another. Reviewing your table of available routes is like reading a map.Picture yourself visiting Nairobi National Park in Kenya for the first time. When you land at Jomo Kenyatta International Airport, you take a taxi, and the taxi driver can use different routes to get you to Nairobi National park. Many routes are available, but you want to take the fastest or most efficient path to your destination. You can’t make it to your destination if there's no available route. Similarly, if a suitable route doesn’t exist in your VPC routing table, then your computer (IP address) in one VPC can’t communicate with computers (IP addresses) in the connected VPC.AWS security groupsA security group in AWS is a virtual firewall that helps control inbound and outbound traffic into and out of AWS VPC. A security group can only be linked to one VPC, but you can set up more than one security group in a VPC. For simplicity, we encourage users to create a new AWS security group specific to Timescale VPC peering.Note:There is a limit of five AWS security groups in a VPC. If you’ve hit this limit in your AWS VPC, you can edit an existing security group instead of creating a new one.Using a city as an analogy, your IP address would be your postal address; the subnet would be your city and country code, and CIDR would be your area code. Via routing, we pick the desirable route from one IP address to another.Configuring Your Networks for VPC PeeringVPC peering involves two networks which, if not carefully designed, can lead to a conflict between the two VPCs network IP addresses—for instance, assuming that your AWS VPC CIDR block is using a network such as 10.0.0.0/16, which has a subnet mask of 255.255.0.0 and a usable host IP range of 10.0.0.1 - 10.0.255.254 within the AWS environment. You must ensure that there is no overlap with this range of IP addresses on your Timescale VPC CIDR block.The image below illustrates a CIDR overlap error when using AWS VPC CIDR block of 10.0.0.0/16 and Timescale VPC CIDR block of 10.0.0.0/16. The error displayed in red reads, “AWS vpc peering connection request has failed.” The error will not indicate that this is a CIDR overlap; therefore, part of the troubleshooting process whenever the VPC creation errors out should be to check both the AWS and Timescale VPC CIDR blocks to ensure the two CIDR blocks do not overlap.If you see this error message in Timescale, you're looking at a CIDR overlapThe best way to avoid overlap is to be sensible with the IP address range you assign to each side of the VPC. Estimate the total number of client devices or servers connecting to the TimescaleDB database now and project future usage based on your company's growth.We gave an example of a network with a subnet mask of /24, giving us 254 host IP addresses. In most cases, this number of host addresses is enough to serve most organization requirements, but you can also use a lower subnet mask (e.g., /20 to increase the number of host IP addresses).Using AWS VPC CIDR block 10.0.0.0/16 and a Timescale VPC CIDR block of 192.168.0.0/24, the VPC peering connection will be successful, as illustrated below. A point worth noting is that the peering connection request will be in a “processing” state on the Timescale console until you accept the peering connection request on the AWS end.Note:The peering connection expires after seven days. If you have not accepted or declined this peering connection request, you will need to initiate the peering connection again from the Timescale end.The screenshot below illustrates a peering connection request's “processing” state.Your peering connection request is processed in TimescaleAccepting a VPC peering request1. Proceed to the AWS VPC peering connections page. The new Timescale peering connection request status will be in “pending acceptance” state.a. Before accepting a new peering connection request, validate it to ensure that it corresponds to your Timescale request.b. You should check fields such as “requester CIDRs” and “requester region” and make sure that these two fields correspond to your Timescale VPC details.2. Click on the peering connection ID hyperlink that has a ""pcx-"" prefix to view more details of this peering connection request.Peering connections screen in AWS3. Go to the Actions drop-down menu and select the “Accept request” option.a. This establishes a VPC peering connection between Timescale and AWS.b. The status on AWS changes to “Active,” while the status on the Timescale end changes to “Accepted.”RoutingWe can now create new Timescale services or migrate existing services into this VPC. However, these services will not be accessible from the public internet or AWS network because we have not yet defined routes to route traffic between AWS and Timescale.Note: Timescale VPC is region-based, and existing services must be in the same region to allow migration of these services to a VPC. This also applies when you create a new service. Timescale has implemented control measures to prevent users from attaching services to a VPC that is in a different region.We defined our AWS VPC CIDR block as 10.0.0.0/16, which means that all AWS services (EC2 instances, Lambda functions, bastion host, QuickSight, etc.) hosted in this VPC will have IP addresses within this CIDR block range, and traffic from these services will need a route to navigate to the Timescale VPC.When an AWS VPC is created, the main route table is created by default. This route table has one route that allows internal traffic to flow between services created within this VPC. We leverage this main route table to define another route that will allow both outbound and inbound traffic to and from Timescale VPC.On the AWS console, under the VPC page, go to the “Route tables” page. All routes are displayed, which also includes routes for other VPCs within your AWS environment. To narrow down the routes associated with your VPC, you can filter route tables using your AWS VPC ID.Route tables page in AWSWe then click on the main route table (that has a “YES” under the “Main” column) and add our Timescale VPC peering connection. The destination column requires our Timescale VPC CIDR block (192.168.0.0/24), and the target column will be the peering connection ID (that starts with the prefix ""pcx-"").Configuring the route in AWSThis newly added route will simply ensure that any traffic destined to or from the Timescale CIDR block 192.168.0.0/24 will be routed through the VPC peering target connection (pcx-).Security groupsSo far, we have created a peering connection between Timescale and AWS and defined a route for traffic flow. Still, by default, all Timescale services ports are blocked, and no traffic can flow from AWS to Timescale. We, therefore, need to open port 5432 explicitly.Why port 5432? All Timescale VPC-peered services are automatically assigned port 5432. The port might be seen as the gate to the house (IP address). You can get super close but can’t get to the door without the gate (port) being open.How-To:Add a new outbound rule of “type” Custom TCP and “port range” 5432, choose custom “destination,” and add Timescale CIDR block (192.168.0.0/24).Creating a security group in AWSPeering Timescale With AWS EC2, Lambda, and QuickSightTo access Timescale services, we need to connect from within the AWS environment. This includes such services as EC2 instances, a bastion host, Lambda functions, QuickSight, and so on.Many corporations have VPN services running in their data centers. You can leverage your organization’s VPN to access the Timescale VPC-peered services directly from your client network. This requires your networking team to set up site-to-site VPN tunnels between your on-premise data center and AWS VPC. You can also leverage AWS direct connect services to link on-premise data centers with AWS VPC. In case you are using your home network, you can also set up a VPN tunnel using other free VPN providers, such as OpenVPN, Wireguard, etc.Now that you’re familiar with the networking concepts necessary to navigate VPC peering and you have your networks ready, let's discuss how to configure three of the most common AWS services, starting with EC2 instances (the most common scenario) to continue with more advanced use cases like Lambda functions and AWS QuickSight.EC2 instanceStart byCreating a new EC2 instancein your AWS VPC (make sure that you use the AWS VPC we used in the above steps), attach the security group we created above, and install PostgreSQL. Below are simplified steps to install PostgreSQL version 13 on an EC2 instance:Step 1:Connect to the EC2 instance and update all installed packages to the latest versions.sudo yum -y updateStep 2:Add PostgreSQL 13 Yum repository.sudo tee /etc/yum.repos.d/pgdg.repo<<EOF
[pgdg13]
name=PostgreSQL 13 for RHEL/CentOS 7 - x86_64
baseurl=https://download.postgresql.org/pub/repos/yum/13/redhat/rhel-7-x86_64
enabled=1
gpgcheck=0
EOFStep 3:Update the packages index file.sudo yum makecacheStep 4:Install PostgreSQL.sudo yum -y install postgresql13 postgresql13-serverStep 5: Initialize the database.sudo yum -y install postgresql13 postgresql13-serverStep 5:Enable and start PostgreSQL Service.sudo systemctl start postgresql-13
sudo systemctl enable postgresql-13Step 6:Check the PostgreSQL service status.sudo systemctl status postgresql-13If the service is active, your PostgreSQL client is ready to connect to Timescale VPC-peered service. To establish a connection to the Timescale VPC-peered service, we need a connection string that includes the following components: Hostname, Database name, Username, Port, and Password.The Timescale service console overview page provides these details except the password. On the Timescale overview page, we have a service URL configured with all these components that we will use to connect to our service.Service connection info in TimescaleGrab the service URL and type the following on your EC2 terminal. Replace theservice_idandproject_idplaceholders with your service URL details:psql -x postgres://tsdbadmin@service_id.project_id.tsdb.cloud.timescale.com:5432/tsdb?sslmode=requireThis command will prompt for a password—provide your VPC-peered Timescale service password and press enter. Congratulations! You are now connected to a Timescale VPC-peered service. Take note that sources outside the AWS environment cannot connect to this VPC-peered Timescale service. You can test this scenario using the above psql command on your workstation terminal or other client tools, e.g., pgAdmin, DBeaver, etc.AWS LambdaAWS Lambda service has gained popularity with Timescale customers and is among the most connected AWS services to VPC-peered Timescale services.Customers connecting to a VPC-peered Timescale service often have issues getting the right VPC settings. This section will highlight the key AWS Lambda VPC settings required to establish connectivity to Timescale services.AWS Lambda VPC settingsThe AWS docs thoroughly define the steps forcreating a new Lambda function, and our focus will be to guide you through key Lambda functions' VPC settings. In AWS, create a Lambda function page under Advanced settings, click on the Enable VPC checkbox, select the AWS VPC that we defined above, select Subnet, and finally, the security group we defined above.Enabling VPC peering for your new AWS Lambda functionsFor existing Lambda functions that don’t have VPC configured, select the Lambda function name on the functions list page, click on configuration, select VPC from the list and click the edit button to edit VPC settings. This will take you to the VPC settings, as highlighted in the AWS Lambda VPC Settings section.Enabling VPC for your existing AWS Lambda functionsTesting AWS Lambda functionsUsing Python 3.9 as our source code language, we create a simple AWS Lambda function that connects to our VPC-peered Timescale service and fetches some records from a hypertable. To do this, we will use the Psycopg2 PostgreSQL driver to perform our CRUD (create, read, update, and delete) operations. AWS doesn’t have these third-party libraries in-built; therefore, we need to deploy the psycopg2-binary Python package as an AWS Lambda layer.Deploying psycopg2-binary Python package in AWS Lambda layerStep 1:Download the correct version of psycopg2-binary file frompypi. E.g., if you are using Python runtime 3.9, the download file should be cp39.Step 2:Uncompress the file download in step 1 (this is a .whl file that needs to be unpacked using a wheel package). The uncompressed data generates three folders: psycopg2, psycopg2_binary-2.9.3.dist-info, and psycopg2_binary.libs.Step 3:Create a directory as indicated below and copy all these three folders:mkdir -p ./python/lib/python3.9/site-packages

cp -r psycopg2 psycopg2_binary-2.9.3.dist-info psycopg2_binary.libs ./python/lib/python3.9/site-packagesStep 4:Compress the parent folder.zip -r python.zip ./pythonStep 5:Navigate to the AWS Lambda console, create a new layer, and upload the zip file we generated in step four. TheCompatible runtimesfield is optional, but this should be your Python version and align with the Psycopg2 library files you downloaded in step one. Save these new layer details.Creating a new layer in AWS LambdaStep 6:To add this layer to your AWS Lambda function code, navigate to your Lambda function (in our case, test-function) under the code window, scroll down to the layers section, and click on “Add a layer.”Adding a layer to your AWS LambdaUnder Layer source, choose Custom layers, and select the layer we created above and the corresponding version.Selecting a custom layer in AWS LambdaWe should now be able to use the Pyscopg2 library in our Lambda function and query a hypertable hosted in our VPC-peered Timescale service:AWS Lambda and Timescale are now peeredAWS QuickSightConnecting from AWS QuickSight to a publicly accessible (not VPC-peered) Timescale service requires simple steps of creating a PostgreSQL dataset with the following Timescale service details:Database server: this represents the service hostname.Port number: this represents the service port number.Database name: Timescale has a predefined database name called tsdb.Username: if no custom users/roles are defined, use the default tsdbadmin user.Psycopg2: this represents your database password.One common error you will encounter is when you try to create a QuickSight data source with SSL enabled. Currently, Timescale only has self-signed certificates, while QuickSight does not accept self-signed certificates. To work around this problem, you have to uncheck the Enable SSL checkbox on the QuickSight Create data source page that is checked by default.Make sure you uncheck the ""Enable SSL"" checkbox on the QuickSight ""Create data source"" page. This will be checked by default.Timescale uses the password authentication method SCRAM (Salted Challenge Response Authentication Mechanism) by default, which is more secure than the MD5 (Message-Digest algorithm) method. However, we have a known problem with QuickSight when connecting a username with the SCRAM authentication method to a Timescale database. You will get an “authentication not supported” error using the SCRAM method.If you're using the SCRAM authentication method, you will get this errorThe workaround is to change the Timescale authentication method from SCRAM to MD5. The MD5 method uses a custom, less secure challenge-response mechanism as it provides a potential security gap if one gets access to the server's password hash.To change the password authentication method on theTimescale service console, navigate to the Operations page, and choose the Reset password option. Authenticate using your service account credentials (credentials linked to your service email and not the database credentials). On the Reset service password prompt, key in the same tsdbadmin database user password (or you can change it if you like), then select MD5 as the authentication type and click on the Reset service password button to save these details.Reset service password in Timescale (select MD5 as the authentication type)With these two workarounds, your QuickSight connection will be successfully validated when connecting to public or VPC-peered Timescale services.Quicksight VPC setupOne of the limitations of setting up a Timescale VPC peer with AWS VPC is that the Timescale service is blocked from public internet access. Therefore, this requires one to use a service hosted within the AWS VPC environment.QuickSight is a service that is heavily used by Timescale customers and among the many support cases, notwithstanding the SCRAM authentication method and SSL issues highlighted above. We aim to summarize the key steps you can follow to configure QuickSight VPC peering with Timescale VPC-peered service.First, ensure that QuickSight is running in the same region as the AWS VPC you want to connect to.Step 1: Set QuickSight region. You can change your QuickSight region by clicking on your profile icon at the top right corner section and choosing the region option.Choosing your region in QuickSightStep 2:Create the QuickSight VPC connection.On the profile menu (step one above), select Manage QuickSight and then Manage VPC connection. This opens the Manage VPC connections page. If you already have existing QuickSight VPCs defined, these VPCs will be displayed on this page. Click on Add VPC connection, and this page is displayed:Adding a VPC connection in QuickSightVPC connection name:give a meaningful name to your VPC connection.VPC ID: use your AWS VPC ID.Subnet ID: use a subnet with a route table linked to your VPC Internet Gateway.Security group ID: use the security group we created when peering Timescale service. We just need to modify the inbound rules to include QuickSight IP address range CIDR for our region, as listed onthis site. In our example, we are using eu-west-1 (Ireland) region, and therefore, the IP address CIDR to use is 52.210.255.224/27. Below is a snapshot of how to modify our security group inbound rule using the HTTPS protocol.Editing inbound rules in AWSStep 3:Create data source.Navigate to QuickSight’s home page, select the Datasets option, and then the New dataset. We will use PostgreSQL’s data source, which will allow us to provide the following Timescale service properties:Data source name: this represents a descriptive name of your data source.Connection type: use it to select the VPC that we created in step two.Database server: this is the Timescale service hostname that you will get from theTimescale service console.Service port: this is, by default, 5432 for Timescale VPC-peered services.Database name: by default, Timescale creates a database named tsdb.Username: you can use the default tsdbadmin database user or create a new database user for this connection and grant this new user the required object privileges.Password: this represents the database username password.Creating a PostgreSQL data sourceStep 4: Finally, validate your connection. If the validation is successful, click on Create data source.Congrats, VPC Hero Status Unlocked!If you’ve read this far, you now have the superpower of configuring your VPC peering, enabling secure access to your cloud data without exposing your services to the public internet. With your new superhero invisibility cloak, you should be able to keep your cloud infrastructure as safe and isolated as a self-hosted deployment.If you found this information helpful, give VPC peering a try! Everything should go smoothly if you follow the steps outlined in this post, but if you still have any questions, reach out! Our team of qualified and experienced Support engineers is here to help.If you’re not a Timescale customer yet, you canstart a free 30-day trialtoday and explore all that Timescale has to offer. Every Timescale customer has the right to dedicated supportat no extra cost—this includes you during your trial!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/vpc-peering-from-zero-to-hero/
2022-12-14T14:37:00.000Z,How to Turn Timescale Into an Observability Backend With Promscale,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.(This blog post was originally published in February 2022 and updated in December of the same year to include a different deployment option: installing Promscale using a Helm Chart.)The adoption of modern cloud-native distributed architectures has grown dramatically over the last few years due to the great advantages they present if compared with traditional, monolithic architectures—like flexibility, resilience against failure, or scalability.However, the price we pay is increased complexity. Operating cloud-native microservices environments is challenging: the dynamic nature of those systems makes it difficult to predict failure patterns, leading to the emergence of observability as a practice. The promise of observability is to help engineering teams quickly identify and fix those unpredicted failures in production, ideally before they impact users, giving engineering teams the ability to deliver new features frequently and confidently.A requirement to get the benefits of observability is access to comprehensive telemetry about our systems, which requires those systems to be instrumented. Luckily, we have great open-source options that make instrumentation easier—particularlyPrometheus exportersandOpenTelemetry instrumentation.Once a system is instrumented, we need a way to efficiently store and analyze the telemetry it generates. And since modern systems typically have many more components than traditional ones and we have to collect telemetry about each of these systems to ensure we can effectively identify problems in production, we end up managing large amounts of data.For this reason, the data layer is usually the most complex component of an observability stack, especially at scale. Often, storing observability data gets too complex or too expensive. It can also get complicated to extract value from it, as analyzing this data may not be a trivial task. If that’s your experience, you won’t be getting the full benefits of observability.In this blog post, we explain how you can use Timescale and Promscale to store and analyze telemetry data from your systems instrumented with Prometheus and OpenTelemetry.Integrating Timescale and Promscale: Basic ConceptsThe architecture of the observability backend based on Promscale and Timescale is quite simple, having only two components:The Promscale Connector.This stateless service provides the ingest interfaces for observability data, processing that data appropriately to store it in a SQL-based database. It also provides an interface to query the data with PromQL. The Promscale Connector ingests Prometheus metrics, metadata, and OpenMetrics exemplars using the Prometheusremote_writeinterface. It also ingests OpenTelemetry traces using the OpenTelemetry protocol (OTLP) and Jaeger traces using Jaeger gRPC endpoint. You can also ingest traces and metrics in other formats using the OpenTelemetry Collector. For example, you can use the OpenTelemetry Collector with the desired receivers and export the data to Promscale using Prometheus, Jaeger, and OpenTelemetry exporters.A Timescale service(i.e.,a cloud TimescaleDB database).This is where we will store our observability data, which will already have the appropriate schema thanks to the processing done by the Promscale Connector.Diagram representing the different components of the observability stack, where OpenTelemetry, Prometheus, Promscale, Jaeger, and Grafana are running in a Kubernetes cluster, and the observability data is stored in TimescaleCreating a Timescale ServiceBefore diving into the Promscale Connector, let’s first create a Timescale service (i.e., a TimescaleDB instance) to store our observability data:If you are new to Timescale,create an account(free for 30 days, no credit card required) and log in.Once you’re on the Services page, click on “Create service” in the top right, and select “Advanced options.”A configuration screen will appear, in which you will be able to select the compute and storage of your new service. To store your observability data, we recommend you allocate a minimum of 4 CPUs, 16 GB of Memory, and 300 GB of disk (equivalent to 5 TB of uncompressed data) as a starting point, this supports 50k samples per second. Once your data ingestion and query rate increase, you can scale up this setup as you need it.Hereis the resource recommendation guide for Promscale.Once you’re done, click on “Create service.”Wait for the service creation to complete, and copy the service URL highlighted with the red rectangle in the screenshot below. You will need it later!In Timescale, your service URL will be displayed right after creating your serviceNow that your Timescale service is ready, it is time to deploy the Promscale Connector on your Kubernetes cluster. We will discuss two different deployment options:Installing Promscale using Helm chart. This method requires that you are already running Prometheus, OpenTelemetry, or Jaeger in your Kubernetes cluster. With the Promscale Helm chart, you only need a single command to get started.Installing Promscale manually through a Kubernetes manifest. You can use this option if you are already running Prometheus or OpenTelemetry in your Kubernetes cluster.The above-listed deployment options will require that you manually configure the existing Prometheus, OpenTelemetry, Grafana, and Jaeger tools to connect to Promscale.Installing Promscale Using a Helm ChartIf you are already running Prometheus and/or OpenTelemetry in your Kubernetes cluster, you may prefer to use the Kubernetes manifest below, which will only install the Promscacle Connector. (The Promscale Connector is a single stateless service, so all you have to deploy is the connector and the corresponding Kubernetes service.)Promscale is a Helm chart that makes it simple to install the Promscale Connector.To install the Promscale Helm chart, follow these steps:Add Timescale helm repository.helm repo add timescale https://charts.timescale.com/2.  Update the helm repository.helm repo update3.   Now install the Promscale Helm chart. Through the command below, we are also configuring thetimescale-uri:We will connect the Promscale Connector to Timescale through the service URL you obtained when you created it.helm install promscale timescale/promscale --set connection.uri=<DB-URI>Note: Remember to replace <DB-URI> with the service URL from your Timescale service.4.   Once the installation is complete, you are good to go—Promscale is ready! Jump straight to the Prometheus, Grafana, and Jaeger sections to see how you can access and configure these tools.Install Promscale Using a Kubernetes ManifestIf you are already running Prometheus and/or OpenTelemetry in your Kubernetes cluster, you may prefer to use the Kubernetes manifest below, which will only install the Promscacle Connector. (The Promscale Connector is a single stateless service, so all you have to deploy is the Connector and the corresponding Kubernetes service.)Note: Remember to replace <DB-URI> with the service URL from your Timescale service.---
# Source: promscale/templates/service-account.yaml
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: false
metadata:
  name: promscale
  namespace: default
  labels:
    app: promscale
    app.kubernetes.io/name: ""promscale-connector""
    app.kubernetes.io/version: 0.16.0
    app.kubernetes.io/part-of: ""promscale-connector""
    app.kubernetes.io/component: ""connector""
---
# Source: promscale/templates/secret-connection.yaml
apiVersion: v1
kind: Secret
metadata:
name: promscale
  namespace: default
  labels:
    app: promscale
    app.kubernetes.io/name: ""promscale-connector""
    app.kubernetes.io/version: 0.16.0
    app.kubernetes.io/part-of: ""promscale-connector""
    app.kubernetes.io/component: ""connector""
stringData:
  PROMSCALE_DB_URI: ""<DB-URI>""
---
# Source: promscale/templates/svc-promscale.yaml
apiVersion: v1
kind: Service
metadata:
  name: promscale-connector
  namespace: default
  labels:
    app: promscale
    app.kubernetes.io/name: ""promscale-connector""
    app.kubernetes.io/version: 0.16.0
    app.kubernetes.io/part-of: ""promscale-connector""
    app.kubernetes.io/component: ""connector""
spec:
  selector:
    app: promscale
  type: ClusterIP
  ports:
  - name: metrics-port
    port: 9201
    protocol: TCP
  - name: traces-port
    port: 9202
    protocol: TCP   
---
# Source: promscale/templates/deployment-promscale.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: promscale
  namespace: default
  labels:
    app: promscale
     app.kubernetes.io/name: ""promscale-connector""
    app.kubernetes.io/version: 0.16.0
    app.kubernetes.io/part-of: ""promscale-connector""
    app.kubernetes.io/component: ""connector""
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: promscale
  template:
    metadata:
      labels:
        app: promscale
        app.kubernetes.io/name: ""promscale-connector""
        app.kubernetes.io/version: 0.16.0
        app.kubernetes.io/part-of: ""promscale-connector""
        app.kubernetes.io/component: ""connector""
      annotations: 
        prometheus.io/path: /metrics
        prometheus.io/port: ""9201""
        prometheus.io/scrape: ""true""
    spec:
      containers:
        - image: timescale/promscale:0.16.0
          imagePullPolicy: IfNotPresent
          name: promscale-connector
          envFrom:
          - secretRef:
              name: promscale
          ports:
            - containerPort: 9201
              name: metrics-port
            - containerPort: 9202
              name: traces-port
      serviceAccountName: promscaleTo deploy the Kubernetes manifest above, run:kubectl apply -f <above-file.yaml>And check if the Promscale Connector is up and running:kubectl get pods,services --selector=app=promscaleConfiguring PrometheusPrometheusis a popular open-source monitoring and alerting system used to easily and cost-effectively monitor modern infrastructure and applications. However, Prometheus is not focused on advanced analytics. By itself, Prometheus doesn’t provide durable, highly available long-term storage.One of the top advantages of Promscale is thatit integrates seamlessly with Prometheus for the long-term storage of metrics. Apart from its 100 % PromQL compliance,multi-tenancy, and OpenMetrics exemplars support, Promscale allows you to use SQL to analyze your Prometheus metrics: this enables more sophisticated analysis than what you’d usually do in PromQL, making it easy to correlate your metrics with other relational tables for an in-depth understanding of your systems.Manually configuring Promscale as remote storage for Prometheus only requires a quick change in the Prometheus configuration. To do so, open the Prometheus configuration file and add or edit these lines:remote_write:
  - url: ""http://promscale-connector.default.svc.cluster.local:9201/write""
remote_read:
  - url: ""http://promscale-connector.default.svc.cluster.local:9201/read""
    read_recent: trueCheck outour documentationfor more information on how to configure the Prometheus remote-write settings to maximize Promscale metric ingest performance!Configuring OpenTelemetryOur vision for Promscale is to create a unified interface where developers can analyzeall their data. How? By enabling developers to store all observability data (metrics, logs, traces, metadata, and other data types) in a single, mature, open-source, and scalable store based on PostgreSQL.Getting closer to that vision,Promscale includes beta support for OpenTelemetry traces. Promscale exposes an ingest endpoint that is OTLP-compliant, enabling you to directly ingest OpenTelemetry data, while other tracing formats (like Jaeger, Zipkin, or OpenCensus) can also be sent to Promscale through the OpenTelemetry Collector.If you want to learn more about traces in Promscale, watch Ramon Guiu (VP of Observability at Timescale) and Ryan Booz (former senior developer advocate) chat about traces, OpenTelemetry, and our vision for Promscale in the following stream:To manually configure the OpenTelemetry collector, we will add Promscale as the OTLP backend store for ingesting the traces that are emitted from the collector, establishing Promscale as the OTLP exporter endpoint:exporters:
  otlp:
    endpoint: ""promscale-connector.default.svc.cluster.local:9202""
    insecure: trueNote: In the above OTLP exporter configuration, we are disabling TLS setting insecure to true for the demo purpose. You can enable TLS by configuring certificates at both OpenTelemetry-collector and Promscale. In TLS authentication Promscale acts as the server.To export data to an observability backend in production, we recommend that you always use the OpenTelemetry Collector. However, for non-production setups, you can send data from the OpenTelemetry instrumentation libraries and SDKs directly to Promscale using OTLP. In this case, the specifics of the configuration depend on each SDK and library—see the corresponding GitHub repository or theOpenTelemetry documentationfor more information.Installing Jaeger QuerySince our recent contribution to Jaeger, it now supports querying traces from a compliant remote gRPC backend store and the local plugin mechanism. Now, you can directly use upstream Jaeger 1.30 and above to visualize traces from Promscale without the need to deploy our Jaeger storage plugin.A huge thank you to the Jaeger team for accepting our PR!In Jaeger, you can use the filters in the left menu to retrieve individual traces, visualizing the sequence of spans that make up an individual trace. That is useful to troubleshoot individual requests.Visualizing Promscale traces in the Jaeger UITo deploy the Jaeger query and Jaeger query service, use the manifest below:---
# Jaeger Promscale deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: default
  labels:
    app: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
        - image: jaegertracing/jaeger-query:1.30
          imagePullPolicy: IfNotPresent
          name: jaeger
          args:
          - --grpc-storage.server=promscale-connector.default.svc.cluster.local:9202
          - --grpc-storage.tls.enabled=false
          - --grpc-storage.connection-timeout=1h
          ports:
            - containerPort: 16686
              name: jaeger-query
          env:
            - name: SPAN_STORAGE_TYPE
              value: grpc-plugin
---
# Jaeger Promscale service
apiVersion: v1
kind: Service
metadata:
  name: jaeger
  namespace: default
  labels:
    app: jaeger
spec:
  selector:
    app: jaeger
  type: ClusterIP
  ports:
  - name: jaeger
    port: 16686
    targetPort: 16686
    protocol: TCPTo deploy the manifest, run:kubectl apply -f <above-manifest.yaml>Now, you can access Jaeger:# Port-forward Jaeger service
kubectl port-forward svc/jaeger 16686:16686

# Open localhost:16686 in your browserConfiguring Data Sources in GrafanaAs shown in the following screenshot, to manually configure Grafana, we’ll be adding three data sources: Prometheus, Jaeger, and PostgreSQL.This screenshot shows the three data sources we’ll be configuring in Prometheus: Promscale-PromQL (the Prometheus data source for querying Promscale using PromQL), Promscale-Tracing ( the Jaeger data source for querying traces from Promscale), and Promscale-SQL (the PostgreSQL data source for querying TimescaleDB)Configuring the Prometheus data source in GrafanaIn Grafana navigate toConfiguration→Data Sources→Add data source→Prometheus.Configure the data source settings:In theNamefield, type Promscale-Metrics.In theURLfield, typehttp://promscale-connector.default.svc.cluster.local:9201, using the Kubernetes service name of the Promscale Connector instance. The 9201 port exposes the Prometheus metrics endpoints.Use the default values for all other settings.Note: If you are running Grafana outside the Kubernetes cluster where Promscale is running, do not forget to change the Promscale URL to an externally accessible endpoint from Grafana.Once you have configured Promscale as a Prometheus data source in Grafana, you can create panels populated with data using PromQL, as in the following screenshot:Example of a Grafana dashboard built querying Promscale with PromQLConfiguring the Jaeger data source in GrafanaIn Grafana, navigate toConfiguration→Data Sources→Add data source→Jaeger.Configure the data source settings:In theNamefield, typePromscale-Traces.In theURLfield, typehttp://jaeger.default.svc.cluster.local:16686, using the Kubernetes service endpoint of the Jaeger Query instance.Use the default values for all other settings.Note:The Jaeger data source in Grafana uses the Jaeger Query endpoint as the source, which in return queries the Promscale Connector to visualize the traces: Jaeger data source in Grafana -> Jaeger Query -> Promscale.You can now filter and view traces stored in Promscale using Grafana. To visualize your traces, go to the “Explore” section of Grafana. You will be taken to the traces filtering panel.Exploring traces from Promscale in GrafanaConfiguring the PostgreSQL data source in GrafanaIn Grafana, navigate toConfiguration→Data Sources→Add data source→PostgreSQL.Configure the data source settings:In theNamefield, typePromscale-SQL.In theHostfield, type<host>:<port>, where host and port need to be obtained from the service URL you copied when you created the Timescale service. The format of that URL is postgresql://[user[:password]@][host][:port][/dbname][?param1=value1&...]In theDatabasefield, type the dbname from the service URL.In the User and Password fields, type the user and password from the service URL.Change theTLS/SSL Modeto require as the service URL by default contains the TLS mode as required.Change theTLS/SSL Methodto File system path.Use the default values for all other settings.In the PostgreSQL details section, enable the TimescaleDB option.You can now create panels that use Promscale as a PostgreSQL data source, using SQL queries to feed the charts:Visualizing observability data in Grafana by querying from TimescaleDB using SQLUsing SQL to Query Metrics and TracesA powerful advantage of transforming Prometheus metrics and OpenTelemetry traces into a relational model is that developers can use the power of SQL to analyze their metrics and traces.This is especially relevant in the case of traces. Even if traces are essential to understanding the behavior of modern architectures, tracing has seen significantly less adoption than metrics monitoring—at least, until now. Behind the low adoption were difficulties associated with instrumentation, a situation that has improved considerably thanks to OpenTelemetry.However, traces were also problematic in another way: even after all the instrumentation work, developers realized there is no clear way to analyze tracing data through open-source tools. For example, tools like Jaeger offer a fantastic UI for basic filtering and visualizing individual traces, but they don’t allow analyzing data by running arbitrary queries or in aggregate to identify behavior patterns.In other words, many developers felt that adopting tracing was not worth the effort, considering the value of the information they could get from them. Promscale aims to solve this problem by giving developers a familiar interface for exploring their observability data and where they can use JOINs, subqueries, and all the advantages of the SQL language.In this section, we’ll show you a few examples of queries you could use to get direct value from your OpenTelemetry tracesandyour Prometheus metrics. Promscale is 100 % PromQL-compliant, but the ability to query Prometheus with SQL helps you answer questions that are impossible to answer with PromQL.Querying Prometheus metrics with SQLExample 1: Visualize the metricgo_gc_duration_secondsin GrafanaTo visualize such metric in Grafana, we would build a panel using the following query:SELECT
  jsonb(v.labels)::text as ""metric"",
  time AS ""time"",
  value as ""value""
FROM ""go_gc_duration_seconds"" v
WHERE
  $__timeFilter(""time"")
ORDER BY 2, 1The result would look like this:Grafana panel built with a SQL query graphing Prometheusgo_gc_duration_secondsExample 2: Calculate the 99th percentile over both time and series(pod_id)for the metricgo_gc_duration_secondsThis metric measureshow long garbage collection takes on Go applications. This is the query:SELECT 
   val(pod_id) as pod, 
   percentile_cont(0.99) within group(order by value) p99 
FROM 
   go_gc_duration_seconds 
WHERE 
   value != 'NaN' AND val(quantile_id) = '1' AND pod_id > 0 
GROUP BY 
   pod_id 
ORDER BY 
   p99 desc;And this is the result:A Grafana panel showing the p99 latency of garbage collection for all pods running Go applicationsWant more examples of how to query metrics with SQL?Check out our docs. ✨Querying OpenTelemetry traces with SQLExample 1: Show the dependencies of each service, the number of times the dependency services have been called, and the time taken for each requestAs we said before, querying traces can tell you a lot about your microservices. For example, look at the following query:SELECT
    client_span.service_name AS client_service,
    server_span.service_name AS server_service,
    server_span.span_name AS server_operation,
    count(*) AS number_of_requests,
    ROUND(sum(server_span.duration_ms)::numeric) AS total_exec_time
FROM
    span AS server_span
    JOIN span AS client_span
    ON server_span.parent_span_id = client_span.span_id
WHERE
    client_span.start_time > NOW() - INTERVAL '30 minutes' AND
    server_span.start_time > NOW() - INTERVAL '30 minutes' AND
    client_span.service_name != server_span.service_name
GROUP BY
    client_span.service_name,
    server_span.service_name,
    server_span.span_name
ORDER BY
    server_service,
    server_operation,
    number_of_requests DESC;Now, you have the dependencies of each service, the number of requests, and the total execution time organized in a table:Extracting service dependencies, number of requests, and total execution time for each API in a service.Example 2: List the top 100 slowest tracesA simple query like this would allow you to quickly identify requests that are taking longer than normal, making it easier to fix potential problems.SELECT
  start_time,
  replace(trace_id::text, '-', '') as trace_id,
  service_name,
  span_name as operation,
  duration_ms,
  jsonb(resource_tags) as resource_tags,
  jsonb(span_tags) as span_tags
FROM span
WHERE
  $__timeFilter(start_time) AND
  parent_span_id = 0
ORDER BY duration_ms DESC
LIMIT 100You can also do this with Jaeger. However, the sorting is done after retrieving the list of traces, which is limited to a maximum of 1,500. Therefore, if you have more traces, you could miss some of the slowest ones in the results.The result would look like the table below:List the slowest traces, including start time, trace id, service name, operation, duration, and tags. You could just click on the trace_id hyperlink for the traces listed in the table to inspect them furtherExample 3: Plot the p99 response time for all the services from tracesThe following query uses TimescaleDB’sapprox_percentileto calculate the 99th percentile of trace duration for each service by looking only at root spans(parent_span_id = 0)reported by each service. This is essentially the p99 response time by service as experienced by clients outside the system:SELECT
    time_bucket('1 minute', start_time) as time,
    service_name,
    ROUND(approx_percentile(0.99, percentile_agg(duration_ms))::numeric, 3) as p99
FROM span
WHERE
     $__timeFilter(start_time) AND
     parent_span_id = 0
GROUP BY time, service_name;To plot the result in Grafana, you would build a Grafana panel with the above query, configuring the data source as Promscale-SQL and selecting the format as “time series”.It would look like this:Graphing the p99 response time for all services in Grafana, querying tracing data using SQLGetting StartedIf you are new to Timescale and Promscale, follow these steps:You can install Promscalehere(it’s 100 % free) or get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.Deploy a full open-source observability stack into your Kubernetes cluster using tobs, our CLI. Apart from Promscale, you will also get tools like Prometheus, Grafana, OpenTelemetry, and Jaeger—fully configured and ready to go. 🔥Check out our documentationfor more information on the architecture of Promscale, Prometheus, and OpenTelemetry. You will also find tips for querying and visualizing data in Promscale.Check out thePromscale GitHub repo. Promscale is open-source and completely free to use. (GitHub ⭐️  and contributions are welcome and appreciated! 🙏)Explore our Youtube Channelfor tutorials on Timescale, TimescaleDB features like compression and continuous aggregates, Promscale, and much more!And whether you’re new to Timescale or an existing community member, we’d love to hear from you!Join us in our Community Slack: this is a great place to ask any questions on Timescale or Promscale, get advice, share feedback, or simply connect with the Timescale engineers. We are 8 K+ and counting!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-turn-timescale-cloud-into-an-observability-backend-with-promscale/
2022-11-28T09:02:56.000Z,re:Invent With Timescale: How We’re Engaging With the Cloud Community,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.One of the best-known tech events in the world is back to inspire the global cloud community, and we couldn’t be more excited to be there as proud bronze sponsors. From November 28 to December 2, all roads (and URLs) lead toAWS re:Inventin Las Vegas, where organizers expect around 50,000 attendees on-site and a few thousand more online.Ready to welcome them—just maybe not all at once—is your very own Timescale Team. As we hit “publish” on this blog post, a group of Timescalers from across the globe and the company (including our co-founders,Ajay KulkarniandMike Freedman) are already at the event, ready totalk about the future of our cloud databaseand answer any questions you may have about Timescale and Promscale.Catch up with our team at The Venetian, on booth 452, for some roaring swag (yes, we’re biased) and open technical conversations about all things time series, databases, and of course, cloud.The Timescale swagLightning Talk at re:Invent: Transform Your Business With Time-Series DataA talk not to be missed by developers and other technologists working to deliver applications in a world ruled by data,“Transform Your Business With Time-Series Data” highlights how time-series data is at the heart of the current data revolution.Blog update: we have uploaded James' talk following AWS re:Invent, so you don't miss out on the time-series revolutionTimescale’s senior director of developer advocacy,James Blackwood-Sewell, shows how time-series data drives innovation around us, from IoT-powered transportation networks to SaaS application analytics and autonomous financial trading algorithms. These innovations deliver continued value to businesses, schools, and communities of all sizes, all over the world.A few use cases for time-series data (sneak preview from James’ slides)If you want to learn how you can leverage time-series data for your business, allowing you to measure change in an evolving world and build powerful and scalable applications with Timescale and AWS, this Lightning Talk is for you.Mark Your Calendar:When: November 30, 4.45 p.m. – 5 p.m. PTWhere:VEN-Lightning, Theater 1Session ID:PRT061The red arrow points to our booth, and the blue square marks the spot for Theater 1, where our senior director of developer advocacy is delivering his talkBooth ActionThe great thing about conferences is not just talking to industry leaders and game-changers but also learning how things work under the hood from those who actually created them.At booth 452, our re:Invent spot at The Venetian(check the map in the previous section for guidance and look out for the signature Timescale dark yellow and blue coloring), you’ll find Timescalers ready to welcome you, hand out fun swag, and learn more about your time-series projects and challenges.And the best part: they’re also more than ready to help you overcome them. For AWS re:Invent 2022, the team has put togetherfive demostargeting a number of business fields so you can see for yourself how Timescale’s products can boost your productivity, deliver an incredible user experience, and save costs for those developing applications in a wide range of industries. And, of course, you’ll have personalized assistance from our team to answer all your questions.The DemosTimescale’s platform featuresFirst things first: we know we keep asking it, but have you met Timescale yet? If you’d like a proper introduction, stop by the Timescale booth. We’ll walk you through the main features of Timescale, showing you how Timescale can help you improve the performance of your time-series applications while keeping your storage bill low.If you want to check out our platform—which you can already do for free bysigning up for a 30-day trial—this is the opportunity to do it in “guided tour” mode, with a Timescale Team member answering your questions and focusing on the most relevant features for your business.In the Timescale demo, the team will walk you through our platform's main featuresDatabase core featuresHaven’t tried Timescale yet? Not a problem! Join us in our booth, and we’ll walk you through the best TimescaleDB features that lend unique superpowers to PostgreSQL. We’ll tell you abouthypertables(automatically partitioned time-series tables),continuous aggregates(automatically refreshed materialized views),thetime_bucketfunction(to quickly build summaries with lower granularities for your dashboards and charts), and much more.Check out some of the TimescaleDB features that lend superpowers to PostgreSQLTimescale for financeFinancial data is time-series data. Many of our customers use Timescale to store and analyze financial and crypto data, including trading tick data, candlesticks, order book data, etc.If this sounds like an application you’re developing, stop by the Timescale booth and tell us about it—in this demo, Timescale’s developer advocateAttila Tothwill show you how you can use Timescale to build a real-time dashboard in Grafana. You’ll learn how to useOHLC hyperfunctions,continuous aggregates, real-time aggregates, andthetime_bucketfunctionto create stunning and speedy visualizations.You can use Timescale and Grafana to store and analyze financial dataTimescale for energyEnergy data is also time-series data. Energy is another big use case for Timescale, but not everything revolves around large company projects: you can use Timescale for your personal projects too!Are you planning to take on your own energy project and need some inspiration? For example, would you like to understand better your energy consumption when installing solar panels? Timescale’s developer advocate,Chris Engelbert,has done precisely that using TimescaleDB. In this demo, Chris will share how he used a Grafana dashboard to track real-time consumption data from his house, all powered by Timescale.Timescale and Grafana come together to track real-time energy consumptionTimescale for observabilityPromscale is a unified storage backend for metrics and tracing data with full Prometheus, Jaeger (we’re Jaeger certified), and OpenTelemetry support. In this demo, developer advocateMathis Van Eetveldewill walk you through monitoring, visualizing, and better understanding your distributed applications using Promscale.This is a simple, accurate, and solid way to understand your systems like never before and really check what’s going on under the hood, with the added benefits of flexible storage, such as configurable downsampling and retention policies (including per-metric retention), data backfilling, and deletion.We start the demo with an overview of our individual service and operation-level statistics, as well as RED (Request, Error, Duration) metrics using nothing but elegant SQL queries (Have we told you that Promscale offers complete PromQL and SQL compliance?).Additionally—because new data visualizations are never enough—we will use Grafana to visualize down and upstream dependency graphs for our individual operations.Promscale offers full support for JaegerDatabase monitoringSpeaking of using Timescale for looking at systems in real-time: what about showing you how we use Timescale and Grafana to monitor our dev environment? In this demo,Kirk Roybalwill share a dashboard monitoring database activity with panels that monitor CPU, network, disk, and memory usage.Time-series data powers all this information: Timescale’s blazing-fast speed allows us to immediately find and respond to issues in real time, like outages, bad-performing queries, regressions in the code, and much more.Monitoring your database using Timescale and GrafanaData tiering to object storeJust a couple of weeks ago, we announced a very exciting functionality in Timescale:our database now includes an object storage layer built on Amazon S3. By running a simple SQL command, you can transparently tier data to the object store for infinite, cost-effective scalability in Timescale. You will pay only for what you store, with no extra charge per query and a baseline price that’s 10x cheaper than our regular storage.As if that wasn’t enough, we’re incredibly proud of the seamless experience it offers developers. This object store is an integral part of Timescale, meaning that you will still interact with a regular PostgreSQL table (or at least, it will feel like it) and query your data with standard SQL, even after it is moved to the object store. If you want to see it in action, stop by our booth! We’ll be happy to share more about this new feature and show you how it works live.P.S.This functionality is also accessible via our private beta. If you want to try it at home,sign up to Timescale (free for 30 days, no credit card required)and navigate to Operations > Data Tiering. Click on the “Request Access” button, and we’ll be in touch soon with the next steps.More Options for Time-Series DevelopersIn an event that’s built around the AWS ecosystem, we expect to find many developers working with heavy data workloads that may have tried other alternatives, such as Amazon RDS for PostgreSQL. If you’re curious about how Timescale compares to RDS, start byreading our latest benchmark.Timescale runs on AWS, offering hosted PostgreSQL with added time-series superpowers. A transition from RDS would surely be swift, but we’re also happy to discuss other challenges you may be facing with your database (be it RDS or others) and help you find the optimal solution for your time-series projects.See You There!AWS re:Invent is one of the sector’s most prominent events, and we’ve put in a lot of time and effort preparing helpful resources to help its attendees perfect their time-series mastery. From demos to swag to a friendly technical team excited to answer all your questions, we have you covered.And in the meantime, if you want to learnhow to do more on AWS using Timescale, check out this blog post—it’s full of hands-on tutorials, blog posts, and videos so you can build time-series applications faster.So that’s it! Now you know what we have in store at re:Invent. Don’t forget to drop by booth 452 at The Venetian and say “hi” to the team—we’ll be ecstatic to see you there! 👋Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/re-invent-with-timescale-how-were-engaging-with-the-cloud-community/
2023-08-31T14:16:35.000Z,Making PostgreSQL Backups 100x Faster via EBS Snapshots and pgBackRest,"If you have experience running PostgreSQL in a production environment, you know that maintaining database backups is a daunting task. In the event of a catastrophic failure, data corruption, or other form of data loss, the ability to quickly restore from these backups will be vital for minimizing downtime. If you’re managing a database, maintaining your backups and getting your recovery strategy in order is probably the first check on your checklist.Perhaps this has already given you one headache or two becausecreating and restoring backups for large PostgreSQL databases can be a very slow process.The most widely used external tool for backup operations in PostgreSQL ispgBackRest, which is very powerful and reliable. But pgBackRest can also be very time-consuming, especially for databases well over 1 TB.The problem is exacerbated when restoring backups from production databases that continue to ingest data, thus creating more WAL (write-ahead log) that must be applied. In this case, a full backup and restore can take hours or even days, which can be a nightmare in production databases.When operating our platform (Timescale, a cloud database platform built on PostgreSQL), we struggled with this very thing. At Timescale, we pride ourselves in making PostgreSQL faster and more scalable for large volumes of time-series data—therefore, our customers’ databases are often large (many TBs). At first, we were completely basing our backup and restore operations in pgBackRest, and we were experiencing some pain:Creating full backups was very slow. This was a problem, for example, when our customers were trying to upgrade their PostgreSQL major version within our platform, as we took a fresh, full backup after upgrade in case there was a failure shortly after. Upgrades are already stressful, and adding a very slow backup experience was not helping.Restoring from backups was also too slow, both restoring from the backups themselves and replaying any WAL that had accrued since the last backup. (In Timescale, we automatically take full and incremental backups of all our customers’ databases.)In this blog post, we’re sharing how we solved this problem by combining pgBackRest with EBS snapshots. Timescale runs in AWS, so we had the advantage of cloud-native infrastructure. If you're running PostgreSQL in AWS, you can perhaps benefit from a similar approach.After introducing EBS snapshots, our backup creation and restore process got 100x faster.This significantly improved the experience for our customers and made things much easier for our team.Quick Introduction to Database Backups in PostgreSQL (And Why We Used pgBackRest)If you asked 100 engineers if they thought backups were important for production databases, they would all say ""yes""—but if you then took those same 100 engineers and gave them a grade on their backups, most wouldn’t hit a pass mark.We all collectively understand the need for backups, but it’s still hard to create an effective backup strategy, implement it, run it, and test that it’s working appropriately.In PostgreSQL specifically, there are two ways to implement backups:logical database dumps, which contain the SQL commands needed to recreate (not restore) your database from scratch, andphysical backups, which capture the files that store your database state.Physical backups are usually paired with a mechanism to store the constant stream of write-ahead logs (WALs), which describe all data mutations on the system. A physical backup can then be restored to get PostgreSQL to the exact same state as it was when that backup was taken, and the WAL files rolled forward to get to a specific point in time, maybe just before someone (accidentally?) dropped all your data or your disk ate itself.Logical backups are useful to recreate databases (potentially on other architectures), but maintaining physical backups is imperative for any production workload where uptime is valued. Physical backups are exact: they can be restored quickly and provide point-in-time recovery. In the rest of this article, we’ll discuss physical backups.How are physical backups usually created in PostgreSQL?The first option is using thepg_basebackupcommand.pg_basebackupcopies the data directory and optionally includes the WAL files, but it doesn’t support incremental backups and has limited parallelization capabilities. The whole process is very manual, too. If you’re usingpg_basebackup, you’ll instantly get the files you need to bootstrap a new database in a tarball or directory, but not much else.Tools likepgBackRestwere designed to overcome the limitations ofpg_basebackup. pgBackRest allows for full and incremental backups, multi-threaded operations, and point-in-time recovery. It ensures data integrity by validating checksums during the backup process, supports different types of storage, and much more. In other words, pgBackRest is a robust and feature-rich tool, making it our choice for PostgreSQL backup operations.The Problem With pgBackRestBut pgBackrest is not perfect: it reads and backs up files, causing an additional load on your system. This can cause performance bottlenecks that can complicate your backup and restore strategy, especially if you’re dealing with large databases.Even though pgBackRest offers incremental backups and parallelization, it often gets slow when executing full backups over large data volumes or on an I/O-saturated system.While you can sometimes rely on differential or incremental backups to minimize data (like we do in Timescale), there are situations in which creating full backups is unavoidable. Backups could also be taken on standby, but at the end of the day, you’re limited by how fast you can get data off your volumes.We shared earlier the example of full database upgrades, but we're also talking about any other kind of migration, integrity checks, archival operations, etc. In Timescale, some of our most popular platform features (likeforks,high-availability replicas, andread replicas) imply a data restore from a full backup.Having a long-running full backup operation in your production database is not only inconvenient, it can also conflict with other high-priority DB tasks, affecting your overall performance. This was problematic for us.The slowness of pgBackRest was also problematic when it was time to restore from these backups. It’s very good at CPU parallelization, but when you’re trying to write terabytes of data as fast as possible, I/O will be the bottleneck. When it comes to recovery time objective or RTO, every minute counts. In case of major failure, you want to get that database up as soon as possible.Using EBS Snapshots to Speed Up the Creation of BackupsTo speed up the process of creating fresh full backups, we decided to replace standard pgBackRest full backups with on-demandEBS snapshots.Our platform runs in AWS, which comes with some advantages. Using snapshots is a much more cloud-native approach to the problem of backups compared to what’s been traditionally used in PostgreSQL management.EBS snapshots create a point-in-time copy of a particular database: this snapshot can be restored, effectively making it a backup. The key is thattaking a snapshot is significantly faster than the traditional approach with pgBackRest: in our case, our p90 snapshot time decreased by over 100x. This gap gets wider the larger your database is!How did we implement this? Basically, we did a one-to-one replacement of pgBackRest. Instead of waiting for the pgBackRest fresh full backup to complete, we now take a snapshot. We still wait for the backup to complete, but the process is significantly faster via snapshots. This way, we get the quick snapshot but also the full data copy and checksumming for datafile integrity, which pgBackRest performs.If a user experiences a failure shortly after an upgrade, we have a fresh backup—the snapshot—that we can quickly restore (we’ll cover how we handle restores next). We still take a fresh full backup using pgBackRest (yay for redundancy), but the key difference is that this happens after the upgrade process has been fully completed.If a failure has happened, the service is available to our customer quickly: we don’t have to force them to wait for the lengthy pgBackRest process to finish before being able to use their service again.The trade-offs for adopting this approach were minimal. The only downside to consider is that, by taking snapshots, we now have redundant backups (both snapshots and full backups), so we incur additional storage costs. But what we’ve gained (both in terms of customer satisfaction and our own peace of mind) is worth the price.Combining EBS Snapshots and pgBackRest for Quick Data Restore: Taking Partial Snapshots, Replaying WALSolving the first problem we encountered with pgBackRest (i.e., slow creation of full backups) was relatively simple. We knew exactly when we needed an EBS snapshot to be created, as this process is always tied to a very specific workflow (e.g., performing a major version upgrade).But we also wanted to explore using EBS snapshots to improve our data restore functionality. As we mentioned earlier, some popular features in the Timescale platform rely heavily on restores, includingcreating forks,high-availability replicas, andread replicas,all of which imply a data restore from a full backup.This use case posed a slightly different and more difficult challenge since to restore from a full backup, such a backup needs to exist first, reflecting the latest state of the service.To implement this, the first option we explored was taking an EBS snapshot when the user clicked “Create” a fork, read replica, or high-availability replica, to then restore from that snapshot. However, this process was still too slow for the end user. To get the performance we wanted, we had to think a bit beyond the naive approach and determine a way to take semi-regular snapshots across our fleet.Fortunately, we already had a backup strategy for pgBackRest in place that we chose to mirror. Now, all Timescale services have EBS snapshots taken daily. For redundancy reasons and to verify file checksums, we still take our standard pgBackRest partial backups, but we don’t depend on them.Once the strategy is solved, restoring data from an EBS snapshot mirrors a restore from pgBackRest very closely. We simply chose the corresponding EBS snapshot we wanted to restore—in the cases mentioned above, always the most recent—and then replayed any WAL that has accumulated since that restore point. Here, it is important to note thatwe still rely on pgBackRest to do our WAL management. pgBackRest works great for us here; nothing gets close in terms of parallel WAL streaming.This EBS snapshotting and pgBackRest approach has given us great results so far. Using snapshots for restores has helped improve our product experience, also providing our customers with an even higher level of reliability. Keeping pgBackRest in parallel has given us peace of mind that we still have a traditional backup approach that validates our data as well as snapshots.We’re continually improving our strategy though, for example, by being smarter about when we snapshot—e.g., by looking at the accumulated WAL since the last snapshot to determine if we need to snapshot certain services more frequently. This practice helps improve restore times by reducing the amount of WAL that would need to be replayed, which is often the bottleneck in this process.On Snapshot PrewarmingOne important trade-off with this EBS snapshot approach is the balance between deployment time and initial performance. One limitation of a snapshot restore is that not all blocks are necessarily prewarmed andmay need to be fetched from S3the first time they are used, which is a slow process.To give props to pgBackRest restore, it does not have this issue. For our platform features, our trade-off was between getting the user a running read replica (or fork or high-availability replica) as quickly as possible or making sure it was as performant as possible.After some back and forth, we decided on our current approach on prewarming: we’re reading as much as we can for five minutes, prioritizing the most recently modified files first. The idea here is that we will warm the data the user is actively engaging with first. After five minutes, we then hand the process off to PostgreSQL to continue reading the rest of the volume at a slower pace until it is complete. For the initial warming, we use a customgoroutinethat reads concurrently from files.Backing It UpWe are not completely replacing our pgBackRest backup infrastructure with EBS snapshots anytime soon: it is hard to give up on the effectiveness and reliability of pgBackRest.But by combining EBS snapshots with pgBackRest across our infrastructure, we’ve been able to mitigate its performance problem significantly, speeding up our backup creation and restore process. This allows us to build a better product, providing a better experience to our customers.If you’re experiencing the same pains we were experiencing with pgBackRest, think about experimenting with something similar! It may cost you a little extra money, but it can be very much worth it.We still have work to do on our end: we will continue to iterate on the ideal snapshotting strategy across the fleet to minimize deployment times as much as possible. We are also looking at smarter ways to prewarm the snapshots and more applications for snapshots in general.If any of these problems interest you,check out our open engineering roles(we’re hiring!). And if you are a PostgreSQL user yourself,sign up for a free Timescale trialand experience the result of EBS snapshots in action.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/making-postgresql-backups-100x-faster-via-ebs-snapshots-and-pgbackrest/
2022-09-15T13:52:39.000Z,How We’re Raising the Bar on Hosted Database Support,"If choosing a database in a crowded market is challenging, choosing where you will run said database can prove equally tricky. The options are endless, but it’s pretty common these days to run your database on a hosted service, whether that’s something like Amazon RDS or a database company’s own cloud, likethe one we have at Timescale. Still, one of the most critical aspects to consider when selecting a hosted provider is cloud support. What will your cloud provider do to help ensure your hosted database is successful?The Value of Deep, Consultative SupportFrom the user’s perspective, one should expect a minimum set of services from the hosting provider, from backups to speedy and steady help when the database is down or other platform-related issues arise, etc. But what if your queries are slow? Or what if a feature does not work as expected? Or would you like guidance on how to architect for future growth? You may not have to consider the physical infrastructure when running a cloud database, but these are still real and valid concerns.Here at Timescale, we want you to be successful with your time-series and relational data. One of the ways we do that is through ourSlack Community, where many of us here at Timescale work together with many folks in our community on questions and issues. Many folks from our community work with each other as well. It’s a fantastic, vibrant community and is open to everyone. We also have a very activeTimescale Community Forum. But that’s not all we do—we are raising the bar on hosted database support.For our users on Timescale, we take it one step further. We have a team of support engineers located around the world to help with migrations, data modeling, query or ingest performance, compression settings, and more.We offer deep, consultative support for every Timescale user at no additional charge.This is an investment we’re making in our users, just as our users are making an investment in us by choosing us to host their time-series and relational data. As a comparison, with Amazon RDS,deeply consultative support in addition to general guidance and best practices starts at over $5,000 per month, and lower tiers have only a community forum or only offer general advice.Here’s what you can expect from working with the Timescale Support Team.Every Case Is UniqueAdopting a new technology (or refining the operations of the one you’re already using) is hard. While TimescaleDB is an extension on top of PostgreSQL, making it much more immediately recognizable and reducing the amount of new information to learn, we still have our own lingo and concepts.Hypertablesandcontinuous aggregatesandcompression. Oh, my!The thing about doing support for a product like TimescaleDB is that there often aren’t easy answers. This is actually a good thing! Our users are talented professionals. Because of that, no easy answers means that the things that do have easy answers—basic configurations, API conventions, etc.—are either well documented or intuitive enough that folks have already figured them out.What gets left over are the less clear-cut questions and scenarios: Why did this query perform poorly? What should I use for segmenting in compression based on X, Y, and Z factors? Should I use one hypertable or many hypertables?✨Editor's Note:One of the most common questions is optimizing “chunk” size. (“Chunk” is Timescale lingo for data partitions within a table.)We answered this question in this blog post.These questions have many variables, requiring some back and forth. That is the key to our support here at Timescale: collaboration. We don’t tend just to give rote answers.Because when it comes to data, there is no one-size-fits-all strategy. We work with you to understand your problem and requirements and devise a solution that works for you in your specific circumstances.Our approach is inquisitive and exploratory because we want to answer not only the question you have now but the ones you don’t know you have yet.Work With Your Customers, Improve Your ProductWe believe strongly in working through questions and answers together. We’re all subject to the unknown unknowns of theJohari window, so by working together, we collectively increase the things we know and decrease the things we don’t. We understand that when you’re looking for help, the last thing you probably want to get back is a bunch of questions. We ask them to gain a more holistic understanding of the issue at hand to provide more than a bandaid but a real, lasting solution.Through our interactions with individual customers, we improve our product for everyone.We view every support case as an opportunity to learn, both as individuals and as a company. As a company, your cases tell us where we can improve. If something isn’t working, that’s something we should look into making better. As individuals, we learn new and innovative ways to look at, analyze, collect, and use data. We learn about operational models and DevOps practices. We learn about new and exciting technologies that work alongside our own that help you answer the questions you need to ask of your data.In support, at our core, we are eternally curious. Working with you gives us the opportunity to explore new and different things every day. Along the way, we hope we can both learn something as we work through your questions together. At the end of the day, this interchange of ideas makes us all better.Work With Us!We’d love to work through some of your questions together. If you are a current user ofTimescale, you know where we’re at—shoot us a message at[email protected].If you’re not yet a Timescale user,you can use it for free for 30 days, no credit card required—and you have full access to our support team during your trial. We look forward to working with you!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-were-raising-the-bar-on-hosted-database-support/
2022-10-04T14:08:19.000Z,13 Tips for Better Data Aggregation in PostgreSQL and TimescaleDB,"One of the main challenges of working with time-series data is effectively running aggregations over high data volumes. In PostgreSQL, you can retrieve data aggregations using various methods:Querying your data directly with an aggregation function and aGROUP BY. You will probably find this option slow if you’re aggregating over large data volumes.Querying aviewthat in turn calls an aggregation function with theGROUP BY. Having your query saved as a view is handy, but it won’t actually improve your query latency since a view is a simple alias for your original query.Querying a PostgreSQLmaterialized viewof cached aggregates, also known as materialized aggregations. This will make your aggregate queries run faster since the materialized view will store previously computed results.However, you may not get up-to-date results.As you can see, these options have shortcomings and may not get the job done—cue in TimescaleDB’s continuous aggregates. Continuous aggregates allow you to materialize aggregations well ahead of time so that your application can quickly retrieve the cached aggregates without waiting for them to be computed at query time.That leads us to the fourth data aggregation method:4.Querying acontinuous aggregate: when you query a TimescaleDB continuous aggregate, it combines cached aggregated values from materialized aggregations with thenewerdata that has not been materialized. It provides an efficient and low-impact mechanism to refresh materialized aggregates more frequently for up-to-date results.In this blog post, we’ll give you helpful tips on how to set up and work with continuous aggregates (we lovingly call them “caggs” at Timescale). This advice is based on our experience as Timescale Support engineers helping numerous customers get started with continuous aggregates. Haven’t tried continuous aggregates yet?Start your free Timescale account now—we look forward to working with you!A Refresher on Continuous AggregatesMaterialization in PostgreSQL allows you to pre-compute aggregations over your data and makes them available as cached values. This provides snappy response times to your application when querying these cached values.But while materialized views are commonplace in PostgreSQL, refreshing one after data insert, update, or delete is often a compute-intensive exercise that materializes aggregates for the full view. This process unnecessarily consumes additional CPU and memory resources for underlying data that has not changed and may leave the materialized view unavailable to your application.With continuous aggregates, Timescale introduced a far superior method to speed up your application queries for aggregates. This method continuously and intelligently refreshes your materialization for new inserts and updates to the underlying raw data. It provides more flexibility over how soon your materialized aggregates are refreshed while keeping your continuous aggregates available.For a deeper dive into the inner workings of materialization and continuous aggregates, readHow PostgreSQL Views and Materialized Views Work and How They Influenced TimescaleDB Continuous Aggregates.Although powerful, it’s worth mentioning that continuous aggregates come with some limitations. For example, they require atime_bucketfunction, and you can only query one hypertable in your continuous aggregate definition query.Here is a simple example of a continuous aggregate definition that we will reference, when needed, throughout the blog.This is basedon ourdevice opsmedium-sized sample dataset.CREATE MATERIALIZED VIEW device_battery_daily_avg_temperature
WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 day', time) as bucket_day,
    device_id,
    avg(battery_temperature) as battery_daily_avg_temp
FROM readings
GROUP BY device_id, bucket_day
WITH NO DATA;You can refer to our documentationfor more details on creating continuous aggregates and their limitations. Now, let’s jump to the tips.Tip 0: Don't Forget to Create a Refresh Policy!Have you ever found yourself in the following scenarios?You created a continuous aggregate, and your query’s performance has not improved.You created a continuous aggregate, and initially, your query’s performance is great, but it degrades over time.You created a continuous aggregate, and your materialized aggregates have progressively become inaccurate.If this sounds familiar, it is very likely that you do not have a continuous aggregate refresh policy (a.k.a. refresh policy) or that the policy has yet to run.Look for any policy job using the following query:select * from timescaledb_information.jobs;Look for policy execution using the following query:select * from timescaledb_information.job_stats;Create a refresh policy if it does not exist:SELECT add_continuous_aggregate_policy('device_battery_daily_avg_temperature',
 	start_offset => INTERVAL '3 days',
 	end_offset => INTERVAL '1 hour',
 	schedule_interval => INTERVAL '1 day');Tip 1: Remember That Your Refresh Policy Materializes Aggregates for Time Buckets Betweennow()and the PastThe continuous aggregate policy will only materialize aggregates for time buckets that fit entirely within the refresh time window.now()is the latest possible time for a continuous aggregate policy refresh time window. Therefore, the refresh policy will only materialize aggregates for time buckets beforenow().Tip 2: You Can Manually Materialize Aggregates for Any Time Bucket—Past, Present, and FutureUsingrefresh_continuous_aggregate, you can manually materialize aggregations for any time bucket, past, present, and future. To do this, specify a refresh time window that fully overlaps with the desired time bucket.For instance, if you have a continuous aggregate for daily temperature averages, you can materialize today's averages (even if today’s bucket is still receiving new data) by defining a refresh time window that starts at/before midnight today and ends at/after midnight tomorrow.So hypothetically,if today’s date is November 16, 2016, manually materializing aggregates for that date would look as follows:CALL refresh_continuous_aggregate('device_battery_daily_avg_temperature',
	window_start => '2016-11-16 00:00:00+00',
	window_end => '2016-11-17 00:00:00+00');Tip 3: Aim to Create a Continuous AggregateWITH NO DATA...As a general recommendation from Support, we advise that you create a continuous aggregate with theWITH NO DATAoption. By default, creating a continuous aggregate without that option materializes aggregates across the entire underlying raw data hypertable, regardless of when the cagg creation occurred.Instead, let the continuous aggregate policy do the job for you. And if your underlying raw data hypertable has historical data older thanstart_offsetin your cagg policy, you can manually and progressively refresh it usingrefresh_continuous_aggregate.Note that you can have filters on time in your continuous aggregate query definition. This means you can limit which time buckets will materialize by defining the underlying raw data over which to aggregate. This is very rare since a continuous aggregate is expected to continuously materialize time buckets for newer underlying raw data.Tip 4: ...But, At First, It Will Compute Aggregates at Query Time for Any Time Bucket Over the Entire Underlying Raw Data HypertableContinuous aggregates come in two variations:materialized_onlyandreal-time.  By default, continuous aggregates are real-time aggregates. The latter allows you to query and compute in real-time aggregates over newer time buckets (in time, that is) that have not been materialized.Interestingly, the continuous aggregate determines the data that needs to be aggregated in real time based on a watermark for the most recently materializedtime_bucket. Therefore, a newly created continuous aggregateWITH NO DATAhas a watermark that is eitherNULLor set to a time (i.e.,4714-11-24 00:00:00+00 BC) way back in the past. We hope you don’t happen to be collecting data with prehistoric timestamps before that watermark.Long story short, you can query your cagg for aggregates on anytime_bucketover your entire underlying raw data hypertable. But the aggregates are computed in real time as nothing is yet materialized.Checking the watermark:Get the cagg ID.SELECT id from _timescaledb_catalog.hypertable
	WHERE table_name=(
    	SELECT materialization_hypertable_name
        	FROM timescaledb_information.continuous_aggregates
        	WHERE view_name='device_battery_daily_avg_temperature'
	);Use the cagg ID to get its current watermark.SELECT COALESCE(
	_timescaledb_internal.to_timestamp(_timescaledb_internal.cagg_watermark(17)),
	'-infinity'::timestamp with time zone
);So, what happens when you have a year’s worth of underlying raw data and create a continuous aggregate with daily time bucketsWITH NO DATAaccompanied by a cagg policy that materializes/refreshes aggregates for the last three months of your data?Before any time buckets are materialized, you will be able to query the daily aggregate all the way back to the earliest underlying raw data point. As soon as the continuous aggregate policy executes its first run, the daily aggregations now available to your query will only go as far back as three months from the time of this first policy run.So, if you ever find yourself wondering why your continuous aggregate data disappeared partially or in full, check again.Tip 5: A Materialized Aggregate Over a Time Bucket  Is No Longer Recomputed at Query TimeOnce aggregations for any given time bucket are materialized, their values will not change until the next refresh, at the very earliest. Therefore, materialized aggregates may get out of date, especially for ingest workloads with backfills. However, they remain current for append-only ingest workloads.Depending on how quickly your materialized aggregates get out of date due to incoming data backfills, you may need to set up your refresh policy to run more frequently.For append-only workload, schedule your refresh policy to run once per day.SELECT add_continuous_aggregate_policy('device_battery_daily_avg_temperature',
 	start_offset => INTERVAL '3 days',
 	end_offset => INTERVAL '1 day',
 	schedule_interval => INTERVAL '1 day');For backfill workload, schedule the refresh policy to run multiple times per day (or as frequently as necessary based on your application requirements).SELECT add_continuous_aggregate_policy('device_battery_daily_avg_temperature',
 	start_offset => INTERVAL '3 days',
 	end_offset => INTERVAL '1 day',
 	schedule_interval => INTERVAL '30 minutes');Tip 6: Thus, Choose Your Refresh Policy WiselyThere are many factors to consider when choosing your continuous aggregate refresh policy schedule:The accuracy requirements for materialized aggregationsWhether your ingest workload includes backfillThetime_bucketinterval in the continuous aggregate’s query definitionWith an append-only ingest workload (a.k.a. data ingested with increasing timestamps), refreshing existing materialized aggregations is essentially a no-op. Therefore, the schedule interval neednotbe shorter than thetime_bucketinterval. Additionally, the most recenttime_bucketwill not be materialized until(now() - end_offset)is greater or equal to midnight the following day.And then, when we add backfill ingest workload, the accuracy of caggs’ materialized aggregations and how soon they should be refreshed for your application becomes essential.The materialized aggregation for any giventime_bucketimmediately goes out of date as soon as new data is ingested into thetime_buckettime window. As such, your application’s requirements on how soon out-of-date materialized aggregations should be refreshed determine the policy schedule.Tip 7: Save Storage by Dropping Your Raw Data While Keeping Your Materialized DataOnce you have materialized your aggregation, do you still need the underlying raw data? Sometimes, the answer is “no.” There are numerous use cases that rely on visualizing data and primarily rely on continuous aggregates to retrieve and display data at various rollups quickly.For instance, we may visualize device battery temperatures for the last day using raw data, show temperatures for the last month using daily averages, and finally, show temperatures for the last year using weekly averages. In this case, raw data only needs to be kept in the hypertable for up to a week, just enough time to compute the aggregates for the last week for the weekly continuous aggregate.Creating a continuous aggregate and refresh policy for device battery daily average temperature:CREATE MATERIALIZED VIEW device_battery_daily_avg_temperature
WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 day', time) as bucket_day,
    device_id,
    avg(battery_temperature) as battery_daily_avg_temp
FROM readings
GROUP BY device_id, bucket_day
WITH NO DATA;


SELECT add_continuous_aggregate_policy('device_battery_daily_avg_temperature',
 	start_offset => INTERVAL '3 days',
 	end_offset => INTERVAL '1 hour',
 	schedule_interval => INTERVAL '1 day');Creating a continuous aggregate and refresh policy for device battery weekly average temperature:CREATE MATERIALIZED VIEW device_battery_weekly_avg_temperature
WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 week', time) as bucket_week,
    device_id,
    avg(battery_temperature) as battery_weekly_avg_temp
FROM readings
GROUP BY device_id, bucket_week
WITH NO DATA;


SELECT add_continuous_aggregate_policy('device_battery_weekly_avg_temperature',
 	start_offset => INTERVAL '2 weeks',
 	end_offset => INTERVAL '1 hour',
 	schedule_interval => INTERVAL '1 week');Create a retention policy on the ‘readings’ hypertable to drop the raw data after at least a week (using three weeks to avoid any boundary issues, so we don’t drop the data before the weekly policy has materialized the aggregate over the most recent week). Here's how:SELECT add_retention_policy('readings', INTERVAL '3 weeks');As a note of caution, one must also take special care when revising the continuous aggregate policy or manually refreshing the continuous aggregate. Refreshing a continuous aggregate’s time window for which raw data has been deleted will result in the continuous aggregate either losing data fortime_bucketswith no underlying raw data or inaccurate data fortime_bucketswith partially deleted underlying raw data.Tip 8: And for Further Savings, Only Keep the Materialized Data You NeedTo expand on tip #7, we can also drop previously materialized aggregates from your continuous aggregate once the data is no longer needed.Retention policy on the daily average temperature cagg to drop data after a month:SELECT add_retention_policy('device_battery_daily_avg_temperature', INTERVAL '1 month’');Retention policy on the weekly average temperature cagg to drop data after a year:SELECT add_retention_policy('device_battery_weekly_avg_temperature', INTERVAL '1 year');Tip 9: Then Compress the Materialized Data That You Want to Keep But No Longer Need to RefreshEach continuous aggregation you create will consume additional storage. With very large data, aggregation on smaller time buckets can amount to significant additional storage consumption. Therefore, consider compressing aggregated data older than your continuous aggregates refresh policy’swindow_startparameter value.For more details, see our introduction to compression on continuous aggregate in our blogIncrease Your Storage Savings With TimescaleDB 2.6: Introducing Compression for Continuous Aggregates.Enabling compression on the continuous aggregate:ALTER MATERIALIZED VIEW device_battery_daily_avg_temperature SET (timescaledb.compress = true);Set up the compression policy. Note that the policy’scompress_aftersetting must be greater than the refresh policywindow_start. In our case,compress_afteris set to seven days, while the refresh policywindow_startis set to three days.SELECT add_compression_policy('device_battery_daily_avg_temperature', INTERVAL '7 days');Tip 10: Yes, You Can Change the Continuous Aggregatechunk_time_intervalBy default, when a continuous aggregate is created, the internal hypertable is assigned achunk_time_intervalthat is 10x thechunk_time_intervalof the hypertable with underlying raw data that our continuous aggregate query is aggregating. For instance, a seven-daychunk_time_intervalfor the queried hypertable turns into a 70-daychunk_time_intervalfor the continuous aggregate’s internal hypertable.At continuous aggregate creation, there is no option to set thechunk_time_interval. However, the value for this config parameter can be modified manually on the continuous aggregate’s internal hypertable to a more convenientchunk_time_intervalthat meets your requirements.Finding the internal hypertable for your continuous aggregate:SELECT
  materialization_hypertable_schema || '.' || materialization_hypertable_name
FROM timescaledb_information.continuous_aggregates
WHERE
  view_schema = 'public' and
  view_name = 'device_battery_daily_avg_temperature';Use the name of the internal hypertable in the result from the above query to change the cagg’schunk_time_interval.SELECT set_chunk_time_interval('_timescaledb_internal._materialized_hypertable_17', INTERVAL '2 weeks');Tip 11: More Data to Process, More Time to RefreshThe performance of a continuous aggregate refresh will depend on the amount of data processed. If you create two continuous aggregates that query the same hypertable but with differenttime_buckets, the continuous aggregate with the query on largertime_buckets(i.e., month) will take longer to refresh compared to the continuous aggregate with the query on smallertime_buckets(i.e., day).However, if your data ingestion involves a significant amount of backfilling data with timestamps in the past, it all comes down to the refresh time window rather than thetime_bucket.Tip 12: You Can Create an Index on Your Aggregated ColumnsOh well, if you must, TimescaleDB 2.7 introduced the next iteration of our continuous aggregates in which all aggregated values are finalized.This means that you can now create an indexon an aggregated column which enables you to speed up queries with filters on a continuous aggregate’s aggregated columns.Tip 13: Align Your Continuous Aggregate Time Bucket to Your Time ZoneAbout a year ago, we introduced an experimentaltime_bucket_ngfunction to support monthly and yearly time buckets and time zones. We have since deprecatedtime_bucket_ngand ported most of its functionality to the originaltime_bucketfunction. For additional details, see our blog postNightmares of Time Zone Downsampling: Why I’m Excited About the New time_bucket Capabilities in TimescaleDB.Now, you can create a continuous aggregate and align the time buckets to a specific time zone rather than the default UTC time zone. Note that you will need a cagg per time zone to support multiple time zones. And we hope that each of your cagg’s query definitions will filter out data that does not belong to its time zone.Here’s how you can create a continuous aggregate and align the device battery's daily average temperature to India's standard time:CREATE MATERIALIZED VIEW device_battery_daily_avg_temperature_India
WITH (timescaledb.continuous) AS
SELECT
    time_bucket('1 day', time, 'Asia/Kolkata') as bucket_day,
    device_id,
    avg(battery_temperature) as battery_daily_avg_temp
FROM readings
GROUP BY device_id, bucket_day
WITH NO DATA;Querying the continuous aggregate and shifting bucket times to India’s standard time:SELECT
    bucket_day::timestamptz AT TIME ZONE 'Asia/Kolkata' as India_bucket_day,
    *
FROM device_battery_daily_avg_temperature_India;Start AggregatingWorking with customers every day, the Timescale Support Team has no doubts that continuous aggregates are one of the most loved TimescaleDB features—just read (or watch) what the folks at Density had to say about them. We hope these tips will help you follow some best practices when using continuous aggregates for improved performance and speed.And you know what they say, seeing it is believing it: if you want to test how continuous aggregates can make a difference in your applications,sign up for Timescale—it’s free for 30 days (no credit card required). You’ll get to experiment with caggs (and so much more!), and we’ll be happy to assist you along the way.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/13-tips-for-better-data-aggregation-in-postgresql-and-timescaledb/
2022-09-29T12:57:19.000Z,What’s New in TimescaleDB 2.8?,"TimescaleDB 2.8 is now available onTimescaleand fordownload. This major release includes the following new features:Thetime_buckethyperfunction now supports bucketing by month, year, and time zone, enabling easier time-based queries and reporting.You can now manage refresh, compression, and retention policies for continuous aggregates all in one easy step.Improved performance of bulkSELECTstatements that fetch high volumes of data from distributed hypertables using theCOPYprotocol.Support forON CONFLICT ON CONSTRAINT UPSERTstatements on hypertables, which enables better compatibility with some GraphQL/ PostgreSQL middlewares.Let’s explore these improvements in more detail.Thetime_buckethyperfunction now supports bucketing by month, year, and time zone“While the newtime_bucketfunctionality is rather simple, its impact is immense and simplifies analytical requests with the need of time zone information.”Chris Engelbert, “Nightmares of Time Zone Downsampling: Why I’m Excited About the Newtime_bucketCapabilities in TimescaleDB”Support for time zones and bucketing your data by monthly or yearly intervals using thetime_buckethyperfunctionwere some of the most requested features, and they’re finally available in TimescaleDB 2.8.Developed with thecommunity's helpover the last couple of months, the new implementation initially lived in TimescaleDB’sexperimental namespaceastime_bucket_ng. The functionality has since been improved through testing and feedback and is now generally available as part of thetime_buckethyperfunction. These capabilities simplify analytical requests with the need for time zone information. By providing atime zoneparameter in thetime_buckethyperfunction, developers can now adjust theoriginaccording to the given time zone. That means that those daily, monthly, or yearly boundaries are also modified automatically. When migrating existing code, all your current queries will work just as they did before the change.Using monthly or yearly buckets and specifying a time zone is simple, as illustrated in the query below:SELECT
   time_bucket('1 month', created, 'Europe/Berlin') AS bucket,
   avg(value)
FROM metrics
GROUP BY 1📚Read more aboutimprovements to thetime_buckethyperfunction.Create and manage multiple policies for continuous aggregates in one stepContinuous aggregateshelp developers query large amounts of time-series datamore quickly, and are one of the hallmark features of TimescaleDB. Prior to TimescaleDB 2.8, you could only add one policy at a time, which could be tedious and complicated. Now with TimescaleDB 2.8’s one-step policy for continuous aggregates, you can add, remove, or modify multiple policies of continuous aggregates with a single command, including refresh, compression, and retention policies. Note that this feature is experimental, so we welcome your feedback on how to improve it for production use.Watch thedemo videoor read ourdocumentationto learn more about one-step policy management for continuous aggregates.Performance improvements for distributed hypertables using theCOPYprotocol for high data volume queriesIn TimescaleDB 2.8, we improved the performance of queries that select a high volume of data from distributed hypertables, such asSELECT *.... For this, we started to use the so-calledCOPYprotocol—the subset of the PostgreSQL native protocol used to efficiently transfer data in bulk. We also made our code more efficient and reduced overhead by streamlining how we work with the internal data structures used to hold and transfer the row data.Hypertables now supportON CONFLICT ON CONSTRAINT UPSERTstatementsWith the TimescaleDB 2.8 release,hypertablesnow support theON CONFLICT ON CONSTRAINTclause, which fixes a long-standing compatibility issue with GraphQL/PostgreSQL middlewares like Hasura, Prisma, and Postgraphile.Try TimescaleDB 2.8If you are using Timescale, upgrades are automatic, and you’ll be upgraded to TimescaleDB 2.8 in your next maintenance window.New to Timescale?Start a free 30-day trial, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/whats-new-in-timescaledb-2-8/
2022-11-15T14:00:00.000Z,Do More on AWS With Timescale: 8 Services to Build Time-Series Apps Faster,"You may know Timescale as the developer ofTimescaleDB, the leading time-series and analytics database built on PostgreSQL. We’re also building Timescale, a cloud-native PostgreSQL solution for time series, events, and analytics thatexpands traditional cloud databases' boundariesby combining all the goodness of PostgreSQL with the flexibility of cloud-native architectures. And we’re building it on AWS.With its extensive catalog of services,AWS is the industry leader in public cloud infrastructure—and the default choice for many developers on where to build their projects. One of the advantages of building in AWS is that you can mix and match itswide range of services and toolsto architect your data infrastructure, avoiding the need to create these services from scratch and speeding up development time.Timescale is built to serve developers working with time series, events, and analytics applications in AWS. You can integrate your Timescale databases seamlessly into your existing AWS infrastructure as your all-in-one datastore for relational and time-series data for your applications.But don’t just take it from us. See what one of our customers,SquareRoots, says about building on AWS with Timescale:""Timescale integrated seamlessly into our AWS data pipeline with AWS IoT Greengrass, AWS Kinesis, and AWS Lambda to help power our controlled environment agriculture platform.""Mark Thompson, Senior Infrastructure EngineerSquare RootsA downside to AWS’s extensive range of tools is the paradox of choice. To help solve that problem, we’ll tell you about eight AWS services that Timescale customers love using with Timescale, ranging from tools to ingest data into your database and business intelligence tools to services for low-cost data archiving and tiering.In particular, we’ll cover the pairing of Timescale with the following services:Amazon VPCAWS LambdaAWS IoT Tools: IoT Core and IoT GreengrassAmazon QuickSightAmazon CloudWatchAWS Managed Service for Apache KafkaAmazon S3Let’s get into it!Amazon VPCVirtual Private Cloud (VPC) peeringis a method of connecting separate Cloud private networks. It makes it possible for the virtual machines in the different VPCs to talk to each other without going through the public internet—resembling a traditional network that businesses would previously operate in their own data center but with the benefits of using scalable cloud infrastructure.Amazon VPCis the service that bridges your Timescale databases and the rest of your AWS infrastructure. VPC peering enables you to securely access data stored in Timescale from your existing cloud infrastructure without ever exposing your services to the public internet. More specifically, this service creates a private network “peering” connection between your Amazon VPC(s) and your Timescale VPC(s), making it possible for both to speak to each other without going through the wider Internet.VPC peering using Amazon VPC enables you to establish a private connection between Timescale and other elements of your AWS infrastructure, giving you maximum security and privacyThis is very useful for running a managed database with the utmost privacy. For example, you may be hesitant to use a managed service because you’re concerned about exposing your database to the public internet. VPC peering solves this issue, giving you a private connection between your database and the rest of your AWS infrastructure. With VPC peering, you can enjoy all the benefits of a managed database service in Timescale without compromising on the isolation you’d get in a self-hosted deployment in AWS.VPC peering is useful for simple peer-to-peer connections, but it can also be used for more advanced deployments. For example, you can create multiple Virtual Private Clouds per service, meaning that you could set up separate VPCs for different applications—or your dev, staging, and production environments—each with its own set of security and access control preferences.Finally, it’s worth noting that using VPC peering in Timescale is very inexpensive—it will only cost you $0.030/hr per connection (which comes out to around $20/month).Want to learn more about Amazon VPC and Timescale? The following resources will tell you everything you need to know:[Blog Post] VPC Peering: From Zero to Hero: A comprehensive guide on how VPC peering works and how to set it up in Timescale. This guide also includes information on how to peer Timescale with your own EC2 instance, AWS Lambda, and Amazon QuickSight (more info on these later in this post).[Docs] VPC Documentation: Contains step-by-step instructions for setting up VPC peering in Timescale using Amazon VPC.AWS LambdaAWS Lambdais a popular serverless compute service that lets you run applications without worrying about provisioning or managing the underlying infrastructure. As a user working with AWS Lambda, you define event-based functions that will run your code in response to triggers.As mentioned in the Amazon VPC section, AWS Lambda is one of the services in which you can use VPC peering to access and insert data into your Timescale databases.Being serverless, AWS Lambda is a powerful tool to operate your data pipelines with almost no operational overhead and paying only for what you consume. You can also connect AWS Lambda withAWS API Gatewayto expose your function as an API endpoint or automatically run the function periodically usingAWS EventBridgeorAWS SNS/SQS. It works with Go, Node.js, Java, or Python code.Timescale customers often use AWS Lambda to route time-series data into Timescale—for example, using AWS Lambda together with edge runtimes for IoT likeAWS IoT Greengrass.Furthermore, AWS Lambda can be used for transforming, fetching, and performing other data operations on tables andhypertablesin your Timescale databases.You can connect AWS Lambda to Timescale via VPC peering. The example above shows how you can directly query hypertables in Timescale using AWS Lambda and psycopg2, the popular PostgreSQL database adapter for PythonTo learn more about AWS Lambda and how to use it in your next project, see the resources below:[Tutorials] AWS Lambda Tutorial: Here’s a tutorial that walks you through how to create a data API for Timescale using AWS Lambda and AWS API Gateway, how to pull data from third-party APIs and ingest it into Timescale, and how to continuously deploy your Lambda function using GitHub Actions.[Blog Post] AWS Lambda For Beginners: Overcoming the Most Common Challenges:Useful advice on navigating the trickiest parts of working with AWS Lambda, like adding external dependencies, overcoming the 250 MB package limit in containers, or how to set up continuous deployment.[Blog Post] How to Peer Timescale With AWS Lambda:Navigate to the “Peering Timescale…” section, where you’ll find detailed instructions on establishing a successful connection between AWS Lambda and Timescale.IoT Tools: AWS IoT Core and AWS IoT GreengrassIoT is one of the most popular use cases for customers on Timescale. Here’s how Timescale can be used withAWS IoT solutionsto build stellar IoT applications. We often hear about AWS IoT Core and AWS IoT Greengrass:AWS IoT Coreestablishes a secure, bidirectional connection between your edge devices and your AWS infrastructure in a serverless manner. It supports the most common networking protocols (LoRaWAN, MQTT, and HTTPS), helping you manage your IoT fleet, which can get significantly complex once you start having thousands (or even millions) of devices.Timescale customers often send sensor data from their devices to tools like AWS IoT Core (to help them manage the connection between edge and cloud) and use services like AWS Lambda to store that sensor data in TimescaleAWS IoT Greengrassis an edge runtime that helps you configure your IoT devices faster via pre-built modules and functionality. This service can be useful if you have a large fleet of devices performing some form of edge processing (like Lambdas or machine learning inference), if devices communicate with each other, or if you're operating with disrupted internet connectivity. AWS IoT Greengrass integrates with AWS IoT Core, but it also allows you to directly stream data to services like Amazon Kinesis or Amazon S3.You can use these tools to build your IoT data architecture, storing your sensor data in Timescale. Our customers love using Timescale for IoT because of its performance at scale (think: real-time queries and dashboards over millions of data points), time-series functionality, seamless integration with data visualization tools and end-user systems, and great cost efficiency for high data volumes.If you’re running an IoT use case,make sure you give Timescale a try(it’s completely free for 30 days) while checking out these resources:[Customer Story] How Edeva Uses IoT to Build Smarter Cities:A software architect at Edeva shares how his team collects huge amounts of data from IoT devices to help build safer, smarter cities and how they leverage Timescale’scontinuous aggregationsfor lightning-fast dashboards.[Tutorial] Visualize Geospatial Data Using Timescale and Grafana:IoT use cases often involve both temporal and geospatial analysis. Grafana includes a WorldMap visualization that helps you see geospatial data overlaid atop a map of the world. This tutorial offers step-by-step instructions on building dashboards for time-series and geospatial data, which are common in smart logistics, and fleet management use cases.[Customer Story] How Everactive Powers a Dense Sensor Network:Everactive engineers share how they’re bringing analytics and real-time device monitoring to scenarios and places never before possible. Learn how they’ve set up their data stack, their database evaluation criteria, their advice for fellow developers, and more.Amazon QuickSightAmazon QuickSightis a managed business intelligence (BI) tool that provides both easy-to-use visualizations and dashboarding to get insights from business analytics. It integrates with a wide range of data sources, including Timescale, via its PostgreSQL driver. It also provides a machine learning functionality for pattern and anomaly detection.It’s another popular tool that Timescale Customers love to use with Timescale via VPC peering.Amazon QuickSight is a powerful BI tool that is very popular among Timescale customers (Source:aws.amazon.com)How to get started[AWS resources] Get Started With Amazon QuickSight.If you’re looking for tutorials to help you navigate QuickSight for the first time, you can start with AWS’s collection of demo videos and getting started guides.[Blog post] How to Peer Timescale With Amazon Quicksight:Navigate to the “Peering Timescale…” section to get tips on how to properly peer Amazon Quicksight with Timescale.Amazon CloudWatchTimescale directly integrates withAmazon CloudWatch, so you can directly monitor your Timescale database services. Amazon CloudWatch provides a reliable, scalable, and flexible monitoring solution that’s easy to spin up in minutes, saving developers the burden of managing their own monitoring systems and infrastructure.Most of our customers are using Timescale in production—mission-critical applications require close monitoring of your service metrics to ensure that your database operates efficiently and without interruption. This is where integrating Timescale with monitoring tools like Amazon CloudWatch can be extremely helpful, allowing you to set up alerts on your service metrics to get notified every time your memory surpasses a certain threshold or once your storage starts to get full.Here’s a five-minute video that walks you through integrating Timescale and Amazon CloudWatch in a few simple steps:To learn more, check out the following resources:[Blog post] Monitoring Your Timescale Services With Amazon CloudWatch:In this blog post, we walk you through integrating Timescale and Amazon CloudWatch to monitor your memory, CPU, and storage metrics.[Documentation] ExportTelemetryData to Amazon CloudWatch:Step-by-step instructions on how to export your Timescale database metrics to Amazon CloudWatch.AWS Managed Service for Apache KafkaApache Kafkais a popular real-time event streaming service used for a wide variety of data-intensive applications. You can write your own producers to insert generated data into Kafka topics and subsequently write consumers to subscribe to those topics to receive all newly generated data.TheKafka Connectframework enables you to easily stream data in and out of Kafka to and from other services and software using pre-written connectors. A popular connector is theJDBC connectorwhich allows you to ingest data into PostgreSQL from a Kafka topic. Because each Timescale database is also a PostgreSQL database, you can use this JDBC sink to ingest data into Timescale.Deploying and maintaining a Kafka cluster can be a monumental task requiring intimate knowledge of Kafka,Zookeeper, and various other tools like Kafka Connect. A good alternative is to useAWS Managed Service for Apache Kafka(or MSK for short), a feature of MSK isMSK Connectthat allows you to deploy Kafka Connectors at scale.[Blog post]Building a Kafka Data Pipeline for Time Series With Kafka Connect and TimescaleBuild data pipelines without the hassle of writing and testing your consumers and producers by linking Kafka Connect to our cloud database optimized for time series.[Blog Post] Ingesting Data from Apache Kafka to Timescale:A Timescale community member shares his advice for ingesting data from Apache Kafka into TimescaleDB.[Docs] JDBC Connector (Source and Sink):Documentation covering the JDBC source and sink connectors, which enable you to exchange data between relational databases and Kafka.[Blog post] Getting Started Using Amazon MSK:Step-by-step instructions on creating an MSK cluster, producing and consuming data, and monitoring your cluster's health.Amazon S3Amazon S3, also known as Amazon Simple Storage Service, is a highly scalable cloud object storage service that stores object data within buckets. It’s built to retrieve large volumes of data.Unlike some of the services and tools mentioned above, no integration or dev work is required to use Amazon S3 with Timescale databases—you can tier data from a Timescale database to Amazon S3 right within a Timescale database itself!We recently releaseda consumption-based, low-cost object storage layer in Timescale built on Amazon S3.By running one command on your Timescale database, you can transparently tier data to this Amazon S3 object storage without leaving your database and while retaining access to all your data via standard SQL. In fact, you will keep the abstraction of a single table (a hypertable) that’s now transparently stretched across multiple storage layers (disk and S3), allowing you to scale your time-series data without breaking the bank.Amazon S3 is an important service for developers building cloud-native applications. It’s an object storage service with excellent durability, high availability, and virtually infinite scalability that allows you to store vast volumes of data at a lower cost than other AWS storage services, like EBS, via its consumption-based pricing. S3 is one of the most popular services in AWS (perhaps the most popular), and it’s widely used for data warehousing and archiving.But building, integrating, and operating a separate data warehouse or data lake for your time-series data means more development work, complexity, and costs. With Timescale, moving data from the database to an object store is as simple as running a SQL command. You’ll pay only for what you store—no extra charge per query and no more paying for an upper bound of storage just in case you need it.And the best part is that even when data is tiered, all data remains fully and directly queryable from within your database via standard full SQL—including predicates and filters, JOINs, CTEs, windowing, and everything else you’re used to in PostgreSQL!Note: This feature is still under active development and, thus, not ready for production use. Still, you can test it by requesting access to the private beta. To do so, follow these steps:Log in to Timescale.In your Service screen, navigate to Operations > Data Tiering. Click on the “Request Access” button, and we’ll be in touch soon.You can request access to Timescale’s bottomless, low-cost object storage on Amazon S3 private beta via the Timescale UIGet Started TodayNow it’s your turn! Pick your favorites from the AWS tools and services list and apply them to your next time-series, analytics, or events project.Sign up for Timescale.The first 30 days are completely free (no credit card required).Do you have feedback or suggestions for more AWS tools and services we should cover next? Let us know in the TimescaleCommunity Forumor on Twitter@TimescaleDB.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/do-more-on-aws-with-timescale-cloud-8-services-to-build-time-series-apps-faster/
2023-01-18T16:23:31.000Z,An Incremental Materialized View on Steroids: How We Made Continuous Aggregates Even Better,"Time-series data is often collected at a much higher granularity than is later required for display or historical storage. Having too much data never sounds like a problem until it becomes one for speed and storage reasons.A materialized viewis commonly used to precalculate data for faster access. So, we roll data up (or downsample it) into lower granularity datasets (from minutes to days, for example). Typically, the process continues through multiple stages of rollups, going from seconds to minutes, days, weeks, months, etc.In the past, these several rollups had to be generated from the actual raw dataset. This meant the raw data had to be around for as long as the largest rollup window, likely increasing your storage needs.With the release of TimescaleDB 2.9, we solved this problem by adding support forhierarchical continuous aggregates.Simply put, continuous aggregates on top of continuous aggregates.Materialized View vs. Continuous AggregateContinuous aggregates,which can be described as incremental and automatically updated materialized views, have been part of TimescaleDB for quite a while now. They are one of the most beloved features, enabling users to pre-aggregate data in the background and making it quickly available when necessary.A common use case for continuous aggregates is dashboards, where data is often displayed at a much lower granularity than it was recorded. Imagine a data point like CPU usage, which is recorded at a second’s granularity. It is unlikely to display it at the same granularity level in Grafana.You’d normally use averages or percentiles over the course of a minute or even lower, such as a five-minute window.SELECT
	time_bucket('5 minutes', ""time"") AS ""time"",
	avg(cpu_usage) AS ""avg_cpu_usage""
FROM cpu_usage_metrics
WHERE ""time"" BETWEEN now() - INTERVAL '5 days' AND now()
  AND ""machine_id"" = 42
GROUP BY 1
ORDER BY 1;While you can query this time window directly from the raw data, depending on the amount of data and the ingress granularity, the query may not satisfy your response time requirements or deliver a “laggy” user experience.Pre-calculating the required granularity helps give an instant feel to the dashboard for an amazing user experience.CREATE MATERIALIZED VIEW cpu_usage_metrics_avg_5min
	WITH (timescaledb.continuous) AS
SELECT
	time_bucket('5 minutes', ""time"") AS ""time"",
	""machine_id"",
	avg(cpu_usage) AS ""avg_cpu_usage""
FROM cpu_usage_metrics
GROUP BY 1, 2
ORDER BY 1;Querying the same result data as above is now as simple as any other query:SELECT
	""time"",
	""avg_cpu_usage""
FROM cpu_usage_metrics_avg_5min
WHERE ""time"" BETWEEN now() - INTERVAL '5 days' AND now()
  AND ""machine_id"" = 42
ORDER BY 1;If you need to work with yet another granularity level, let’s say 15 minutes, just create another continuous aggregate with the necessary rollup window, and you’ll be fine. That is, if the raw data is available for rolling up at the internal refresh window, which can lead to issues when you need to roll up data for a monthly time window. All raw data needs to be available at that point in time.Well, not anymore!Added Speed and Storage Savings With Hierarchical Continuous AggregatesWith TimescaleDB 2.9 or later, you can roll up a continuous aggregate from a previous continuous aggregate. That means theFROMclause can reference another continuous aggregate, which wasn’t allowed before.Returning to the 15-minute example, we can now implement a continuous aggregate using the already pre-aggregated five-minute one. For the sake of correctness (since it uses an average, and those can be tricky when using multi-stage averages), let’s slightly change the five-minute continuous aggregate by adding an intermediatesumandcount.CREATE MATERIALIZED VIEW cpu_usage_metrics_avg_5min
	WITH (timescaledb.continuous) AS
SELECT
	time_bucket('5 minutes', ""time"") AS ""time"",
	""machine_id"",
	avg(cpu_usage)   AS ""avg_cpu_usage"",
	sum(cpu_usage)   AS ""sum_cpu_usage"",
	count(cpu_usage) AS ""count_cpu_usage""
FROM cpu_usage_metrics
GROUP BY 1, 2
ORDER BY 1;With that out of the way, the 15-minute continuous aggregate is as simple as the following:CREATE MATERIALIZED VIEW cpu_usage_metrics_avg_15min
	WITH (timescaledb.continuous) AS
SELECT
	time_bucket('15 minutes', ""time"") AS ""time"",
	""machine_id"",
	sum(sum_cpu_usage) / sum(count_cpu_usage) AS ""avg_cpu_usage""
FROM cpu_usage_metrics_avg_5min
GROUP BY 1, 2
ORDER BY 1;As you can see, we don’t use the average function anymore, but the two additional intermediate values to build the average. For other aggregations, it may be easier or more complex depending on the multi-stage aggregation requirements of the algorithm.Anyhow, we end up with pre-aggregated 15-minute slices per machine, just as if we’d calculated it straight from the raw data. The benefit here is that you can already expire and delete the raw data after the initial five-minute window is calculated, dropping the amount of stored data to one-third. Imagine the storage savings with something like a monthly time window.And that’s not only true for the raw data, but every single continuous aggregate can haveits own retention policy, too. Just make sure the data is further aggregated before it's retired and removed.But there is one additional benefit: speed. It’s much faster to average over three values than 900. While it doesn’t make a massive difference at this level, more complex algorithms will be a lot faster based on the number of data points.Diagram example of hierarchical continuous aggregates functionality for a finance use caseAs a quick side note for the termhierarchical; we called the featurehierarchical continuous aggregatessince you may branch out from one continuous aggregate into many. One example would be a continuous aggregate with one-day time slices, which is then aggregated into multiple continuous aggregates, such as seven days, 14 days, one month, etc. The branching can become arbitrarily complex—the limit is your imagination.Try the New Continuous Aggregates With TimescaleDB 2.9Timescale is happy to releasehierarchicalcontinuous aggregateswith TimescaleDB 2.9. It is one of the most requested and wished-for features, and we love to make our users (you!) happy with the functionality that is actually needed.Like always, this is not the only cool new addition to TimescaleDB 2.9. Other features include time zone support fortime_bucket_gapfill(an extension of the time zone support fortime_bucketin 2.8) or fixed schedule support for background jobs. For a complete list, check out theRelease Notes.For Timescale users, upgrades are automatic, and you’ll be upgraded automatically using the next maintenance window.If you are new toTimescale, start your free 30-day trial now, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/an-incremental-materialized-view-on-steroids-how-we-made-continuous-aggregates-even-better/
2022-11-15T14:19:00.000Z,"Timescale vs. Amazon RDS PostgreSQL: Up to 350x Faster Queries, 44 % Faster Ingest, 95 % Storage Savings for Time-Series Data","Since we launched Timescale, our cloud-hosted PostgreSQL service for time-series data and event and analytics workloads, we have seen large numbers of customers migrating onto it from the general-purpose Amazon RDS for PostgreSQL. These developers usually struggle with performance issues on ingest, sluggish real-time or historical queries, and spiraling storage costs.They need a solution that will let them keep using PostgreSQL while not blocking them from getting value out of their time-series data. Timescale fits them perfectly, and this article will present benchmarks that help explain why.When we talk to these customers, we often see a pattern:At the start of a project, developers choose PostgreSQL because it’sa database they know and love. The team is focused on shipping features, so they choose the path of least resistance—Amazon RDS for PostgreSQL.Amazon RDS for PostgreSQL works well at first, but as the volume of time-series data in their database grows, they notice slower ingestion, sluggish query performance, and growing storage costs.As the database becomes a bottleneck, it becomes a target for optimization. Partitioning is implemented, materialized views are configured (destroying the ability to get real-time results), and schedules are created for view refreshes and partition maintenance. Operational complexity grows, and more points of failure are introduced.Eventually, in an effort to keep up, instance sizes are increased, and larger, faster volumes are created. Bills skyrocket, while the improvements are only temporary.The database is now holding the application hostage regarding performance and AWS spending. A time-series database is discussed, but the developers and the application still rely on PostgreSQL features.Does it sound familiar? It’s usually at this stage when developers realize that Amazon RDS for PostgreSQL is no longer a good choice for their applications, start seeking alternatives, and come across Timescale.Timescale runs on AWS, offering hosted PostgreSQL with added time-series superpowers. Since Timescale is still PostgreSQL and already in AWS, the transition from RDS is swift: Timescale integrates with your PostgreSQL-based application directly and plays nicelywith your AWS infrastructure.Timescale has always strived to enhance PostgreSQL with the ingestion, query performance, and cost-efficiency boosts that developers need to run their data-intensive applications, all while providing a seamless developer experience with advanced features to ease working with time-series data.But don’t take our word for it—let the numbers speak for themselves. In this blog post, we share a benchmark comparing the performance of Timescale to Amazon RDS for PostgreSQL. You will find all the details of our comparison and all the information required to run the benchmark yourself using theTime-Series Benchmarking Suite(TSBS).Time-Series Data Benchmarking: A Sneak PreviewFor those who can’t wait, here’s a summary:for a 160 GB dataset with almost 1 billion rows stored on a 1 TB volume, Timescale outperforms Amazon RDS for PostgreSQL with up to 44 % higher ingest rates, queries running up to 350x faster, and a 95 % smaller data footprint.When we ingested data in both Timescale and Amazon RDS for PostgreSQL (using gp3 EBS volumes for both), Timescale was34 % faster than RDS for 4 vCPUand44 %  for 8 vCPUconfigurations.When we ran a variety of time-based queries on both databases, ranging from simple aggregates to more complex rollups through to last-point queries,Timescale consistently outperformed Amazon RDS for PostgreSQL in every query category, sometimes by as much as 350x(you can see all of the results in theBenchmarking section).Timescale used 95 % less diskthan Amazon RDS for PostgreSQL, thanks to Timescale’snative columnar compression, which reduced the size of the test database from 159 GB to 8.6 GB. Timescale's compression usesbest-in-class algorithms, including Gorilla and delta-of-delta, to dramatically reduce the storage footprint.And the storage savings above don’t even consider the effect of theobject store built on Amazon S3that we just announced for Timescale. This feature is available for testing via private beta at the time of writing but is not yet ready for production use.Still, by running one SQL command, this novel functionality will allow you to tier an unlimited amount of data to the S3 object storage layer that’s now an integral part of Timescale. This layer is columnar (it’s based on Apache Parquet), elastic (you can increase and reduce your usage), consumption-based (you pay only for what you store), and one order of magnitude cheaper than our EBS storage, with no extra charges for queries or usage. This feature will make scalability even more cost-efficient in Timescale, so stay tuned for some exciting benchmarks!In the remainder of this post, we’ll deep dive into our performance benchmark comparing Amazon RDS for PostgreSQL with Timescale, detailing our methods and results for comparing ingest rates, query speed, and storage footprint. We’ll also offer insight intowhyTimescale puts up the numbers it does, with a short introduction to its vital advantages for handling time-series, events, and analytics data.If you’d like to see how Timescale performs for your workload,sign up for Timescaletoday— it’s free for 30 days, there’s no credit card required to sign up, and you can spin up your first database in minutes.Benchmarking ConfigurationAs for our previous Timescale benchmarks, we used theopen-source Time-series Benchmarking Suiteto run our tests. Feel free to download and run it for yourself using the settings below. Suggestions for improvements are also welcome: comment onTwitterorTimescale Slackto join the conversation.We used the following TSBS configuration across all runs:TimescaleAmazon RDS for PostgreSQLPostgreSQL version14.514.4 (latest available)No changessynchronous_commit=off(to match Timescale)Partitioning systemTimescaleDB (partitions automatically configured)pg_partman (partitions manually configured)Compression into columnarYes, for older partitionsNot supportedPartition size4h (each system ended up with 26 non-default partitions)Scale (number of devices)25,000Ingest workers16Rows ingested868,000,000TSBS profileDevOpsInstance typeM5 series  (4 vCPU+16 GB memory and 8 vCPU+32 GB memory)Disk typegp3 (16 K IOPs, 1000 MiBps throughput)Volume size1 TBHypertablesare the base abstraction of Timescale's time-series magic. While they work just like regular PostgreSQL tables, they boost performance and the user experience with time-series data by automatically partitioning it (large tables become smaller chunks or data partitions within a table) and allowing it to be queried more efficiently.If you’re familiar with PostgreSQL, you may be asking questions about partitioning in RDS. In the past, we have benchmarked TimescaleDB against unpartitioned PostgreSQL simply because that’s the journey most of our customers follow. However, we inevitably get questions about not comparing usingpg_partman.Pg_partman is another PostgreSQL extension that provides partition creation but doesn’t seamlessly create partitions on the fly: if someone inserted data outside of the currently created partitions, it would either go into a catch-all partition, degrading performance or, worse, still fail). It also doesn’t provide any additional time-series functionality, planner enhancements, or compression.We listen to these comments, so we decided to highlight Timescale's performance (and convenience) by enabling pg_partman on the RDS systems in this benchmark. After all, the extension is considered abest practice for partitioned tablesin Amazon RDS for PostgreSQL, so it was only fair we’d use it.On our end, we enabled native compression on Timescale, compressing everything but the most recent chunk data. To do so, we segmented by thetags_idand ordered by time descending andusage_usercolumns. This is something we couldn’t reproduce in RDS since it doesn’t offer any equivalent functionality.Almost everything else was exactly the same for both databases. We used the same data, indexes, and queries: almost one billion rows of data in which we ran a set of queries 100 times each using 16 threads. The only difference is that the Timescale queries use thetime_bucket()function for arbitrary interval bucketing, whereas the PostgreSQL queries use extract (which performs equally well but is much less flexible).We have split the performance data extracted from the benchmark into three sections: ingest, query, and storage footprint.Ingest Performance ComparisonAs we started to run Timescale and RDS through our 16-thread ingestion benchmark to insert almost 1 billion rows of data, we began to see some amazing wins. Timescale beat RDS by 32 % with 4 vCPUs and 44 % with 8 vCPUs. Both systems had the same I/O performance configured on their gp3 disk, so we kept looking to get to the bottom of why we were winning on busy systems.To test the outcome without any disk I/O involvement, we usedpgbenchto run the following CPU-hungry SQL statement on 8 vCPU machines (using a scale of 1,000 and 16 jobs) and had some more interesting results straight away.SELECT count(*) FROM (SELECT generate_series(1,10000000)) aTimescale was almost twice as fast, returning an average query latency of518 ms, while RDS returned904 ms. This 50 % difference was consistent on both 4 vCPU and 8 vCPU instances.Unfortunately, we can’t look inside the black box that is RDS to see what’s happening here. One hypothesis is that a large part of this difference is because Timescale gives you the exact amount of vCPU you provisionfor PostgreSQL(thanks, Kubernetes!), while Amazon RDS provides you ahost with that many vCPUs.This means that we (Timescale) pay for the operating overhead on Timescale, while on RDS, you (as the user) pay for this. As instances get very busy and processes fight with the operating system for CPU (like for an ingest benchmark or when you’re crunching a lot of data), this becomes a much bigger advantage for Timescale than we had anticipated. As usual, if anybody has any other possible reasons for this difference, please reach out, we’d love to hear from you.Our benchmark shows Timescale not only ingests data faster across the board but also provides more predictable and faster results under heavy CPU load. Not a bad feature when you want to get the most out of your instances.Query Performance ComparisonQuery performance is something that needs to be optimized in a time-series database. When you ask for data, you often need to have it as quickly as possible—especially when you’re powering a real-time dashboard. TSBS has a wide range of queries, each with its own somewhat hard-to-decode description (you can find aquick primer here). We ran each query 100 times on the 4 vCPU instance types (which wasn’t quick in some cases) and recorded the results.When we look at the table of query runtimes, we can see a clear story. Timescale is consistently faster than Amazon RDS, often by more than 100x. In some cases, Timescale performs over 350x better, and it doesn’t perform worse for any query type. The table below shows the data for 4 vCPU instances, but results are similar across all the CPU types we tested (and of course, if your instance is very busy, you could get even better results).When we examine the amount of data loaded and processed by some of the queries with the larger differences, the reason behind these improvements becomes clear. Timescale compresses data into a columnar format, which has several impacts on performance:Timescale compressed chunks group by column, not by row. When a subset of the columns for a table are required, they can be loaded individually, reducing the amount of data processed (especially for thesingle-groupby-query types).When compressed data is loaded from disk, it takes less time, as there is simply less data to read. This is traded off against additional compute cycles to uncompress the data—a compromise that works in our favor, as you can see in the results above.As compressed data is smaller, more of it can be cached in shared memory, meaning even fewer reads from disk (for a great introduction to this, check outDatabase Scaling: PostgreSQL Caching Explainedby our own Kirk Roybal).And just as a reminder, RDS had pg_partman configured for this test. This shows that while Timescale provides efficient partitioning via hypertables, we also provide a lot more than that (353x more in some instances).Storage Usage ComparisonTotal storage size is measured at the end of the TSBS ingest cycle, looking at the size of the database which TSBS has been ingesting data into. For this benchmark on Timescale, all but the most recent partition of data is compressed into ournative columnar format,which uses best-in-class algorithms, including Gorilla and delta-of-delta, to reduce the storage footprint for the CPU table dramatically.After compression, you can still access the data as usual, but you get the benefits of it being smaller and the benefits of it being columnar.Using less storage can mean smaller volumes, lower cost, and faster access (as we saw in the query results above). In the case of this benchmark, we saved 95 %, reducing our database from 159 GB to 8.6 GB. And this isn’t an outlier, we often see these numbers for production workloads at real customers.Beyond Benchmarks: A Closer Look at TimescaleNow that we’ve examined the results of the benchmark, let’s briefly explore some of the features that make these results possible. This section aims to offer insight into the performance comparison above and highlight some other aspects of Timescale that will improve your developer experience when working with time-series data.✨If you’re new to Timescale, you can alsosign up for freeand follow ourGetting Started guide, which will introduce you to our main features in a hands-on way.Hypertables, continuous aggregates, and query planner improvements for performance at scaleTimescale is purpose-built to provide features that handle the unique demands of time-series, analytics, and event workloads—and as we’ve seen earlier in this post, performance at scale is one of the most challenging aspects to achieve with a vanilla PostgreSQL solution.To make PostgreSQL more scalable, we built features likehypertablesand added query planner improvements allowing you to seamlessly partition tables into high-performance chunks, ensuring that you can load and query data quickly.While some other solutions force you to think about creating and maintaining data partitions, Timescale does this for you under the hood, as queries come in with no performance impact. In fact, some of Timescale’s improvements work on tables that don’t even hold time-series data, like SkipScan, whichdramatically improvesDISTINCTqueries on any PostgreSQL tablewith a matching B-tree index.Another problem that comes with time-series data at scale is slow aggregate queries as you analyze or present data.Continuous aggregateslet you take an often run or costly time-series query and incrementally materialize it in the background, providing real-time, up-to-date results in seconds or milliseconds rather than minutes or hours.While this might sound similar to a materialized view, it not only reduces the load on your database but also takes into account the most recent inserts and doesn’t require any management once it’s configured.Hyperfunctions, job scheduling, and user-defined functions to build fasterOnce you have time-series data loaded, Timescale also gives you the tools to work with it, offering over 100 built-inhyperfunctions—custom SQL functions that simplify complex time-series analysis, such astime-weighted averages,last observation carried forwardanddownsampling with LTTP or ASAP algorithms, and bucketing by hour, minute, month and timezone withtime_bucket(), andtime_bucket_gapfill().We also provide abuilt-in job scheduler, which saves the effort of installing and managing another PostgreSQL extension and lets you schedule and monitor any SQL snippet or database function.Direct access to a global, expert support team to assist you in productionIf you’re running your database in production, having direct access to a team of database experts will lift a heavy weight off your shoulders. Timescale gives all customers access to aworld-class team of technical supportengineers at no extra cost, encouraging discussion on any time-series topic, even if it’s not directly related to Timescale operations. You might want some help with ingest performance, tuning advice for a tricky SQL query, or best practices on setting up your schema—we are here to help.As a comparison,deeply consultative support, general guidance, and best practices start at over $5,000 per monthin Amazon RDS for PostgreSQL. Lower tiers have only a community forum or receive general advice. So this means that you need to pay an extra $60,000 a year just for such support on AWS, while you get this for free on Timescale.Native columnar compression and object storage for cost efficiencyCost is one of the major factors when choosing any cloud database platform, and Timescale provides multiple ways to keep your spending under control.Timescale's best-in-classnative compressionallows you to compress time-series data in place while still retaining the ability to query it as normal. Compressing data in Timescale often results in savings of 90 % or more (take another look at our benchmark results, which actually saw a 95 % storage footprint reduction).Timescale also includes built-in features to managedata retention, making it easy to implement data lifecycle policies, which remove data you don’t care about quickly, easily, and without impacting your application. You can combine data retention policies with continuous aggregates to automatically downsample your data according to a schedule.To help reduce costs even further, Timescale offersbottomless, consumption-based object storagebuilt on Amazon S3 (currently in private beta). Providing access to an object storage layer from within the database itself enables you to seamlessly tier data from the database to S3, store an unlimited amount of data, and pay only for what you store. All the while you retain the ability to query data in S3 from within the database via standard SQL.It’s just PostgreSQLLast but not least, Timescale is just PostgreSQL under the hood. Timescale supports full SQL (not SQL-like or SQL-ish). You can leverage the full breadth of drivers, connectors, and extensions in the vibrant PostgreSQL ecosystem—if it works with PostgreSQL, it works with Timescale!If you switch from Amazon RDS for PostgreSQL to Timescale, you won’t lose any compatibility, your application will operate the same as before (but it will probably be faster, as we’ve shown).ConclusionWhen you have time-series data, you need a database that can handle time-series workloads. While Amazon RDS for PostgreSQL provides a great cloud PostgreSQL experience, our benchmarks have shown that even when paired with the pg_partman extension to provide partition management, it can’t compete with Timescale. According to our tests, Timescale can be over 40 % faster to ingest data, up to 350x faster for queries, and takes 95 % less space to store data when compressed.On top of these findings, we offer a rich collection of time-series features that weren’t used in the benchmark. You can speed queries up even further by incrementally pre-computing responses with continuous aggregates, benefit from our job scheduler, configure retention policies, use analytical hyperfunctions, speed up your non-time-series queries with features like Skip Scan, and so much more.If you have time-series data, don’t wait until you hit that performance wall to give us a go. Spin up an account now: you can use it for free for 30 days; no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescale-cloud-vs-amazon-rds-postgresql-up-to-350-times-faster-queries-44-faster-ingest-95-storage-savings-for-time-series-data/
2022-09-01T14:35:29.000Z,Timescale Tips: Migrate Your PostgreSQL Production Database Without Downtime,"✨Welcome to another edition of Timescale Tips, a blog series that aims to help our customers with their most common database-related problems. This time, we’re talking about PostgreSQL database migration. If you missed the previous posts, check out thebest practices to improve your ingest rate in your cloud serviceandhow to optimize your chunk size in your TimescaleDB databases.Data migration is the most painful step in database management and one you can hardly avoid: you’ll need to tackle a database migration at some point if you're in charge of a database. And if you’re managing data at scale (as is often the case with time-series data applications), this migration may be significant volume-wise.A big challenge you’ll face when dealing with mission-critical applications is how to minimize migration downtime. If you research “how to migrate PostgreSQL databases” online, you will find plenty of advice on usingpg_dump/pg_restoreto export your data and import it into another database. That is an okay (and valid) way to do it but not necessarily the best way to migrate a production database.Let’s try a different approach to help manage and possibly relieve that pain.A Different Approach to Data Migration in PostgreSQLIn this post, we’ll present you with another migration option, which we call “ingest and backfill.” This is slightly different than the ‘dump’ and ‘restore’ method, which many folks default to for migration, and offers some key advantages:It will eliminate downtime, taking off your shoulders the stress and angst of a downtime window.Let’s use one of our customers as an example: they operate a cryptocurrency exchange, trading twenty-four-seven across time zones all over the world. They use Timescale  to record the price of all the different cryptocurrencies in real time, along with their user transactions. A significant downtime would mean that this customer could not operate, losing all the revenue from trading and their customers’ trust. Plus, there’s not a “safe time window” in which they could shut down business since the markets are open twenty-four-seven. How to migrate this database? If you have a similar problem, this blog post will help you—you’ll only need a bit of money and a bit of patience to solve this.You will get peace of mind.Using the “ingest and backfill” method, you will be able to compare your new Timescale instance with your soon-to-be legacy production environment, checking for accuracy and consistency. You will flip the switch only when you’re sure your new production service is ready.The Key: Ingest Data in DuplicateLet’s explain what “ingest and backfill” is all about.The “ingest and backfill” migration strategy revolves around sendingduplicate datato your new Timescale instance for a while and then backfilling whatever historical data you need for your daily applications (if any!). In other words: for a defined period, your ingest process(es) will not only send data to your existing PostgreSQL production environment but also to Timescale. (Check out our first Timescale Tips postandthis read on improving INSERT performanceto optimize your data ingestion.)First, migrate your schema.Before anything else, you need to import your schema to Timescale. You can do this in three steps:Dump your schema intoschema_dump.sqlfile:pg_dump -U <SOURCE_DB_USERNAME> -W \
-h <SOURCE_DB_HOST> -p <SOURCE_DB_PORT> \
--schema-only -f schema_dump.sql <DATABASE_NAME>2.  Edit the dump file andremovethe SQL that changes the ownership. For example, remove this type of line:ALTER TABLE public.my_table OWNER TO myadmin;This is required since the table will be owned by the Timescale administratortsdbadminupon restoration of the schema.3.   Restore the dumped data from theschema_dump.sqlfile into your Timescale service:psql -U tsdbadmin -h <CLOUD_HOST> -p <CLOUD_PORT> \
-d tsdb -a -f schema_dump.sqlOnce the schema has been created on Timescale, one can now convert the PostgreSQL table into a hypertable usingcreate_hypertable().Next, convert the PostgreSQL tables to hypertables.Each time-series table (these comprise mainly inserts and have a date/time field) needs to be converted into a hypertable usingcreate_hypertable(). You need to configure a few parameters when creating a hypertable. The first is the table name itself (relation). The second is the time-series column (time_column_name), and finally, thechunk_time_interval, which defines the separate ‘chunks’ in time.Thechunk_time_intervalis an optional parameter defaulting to ‘1 week’, but we highly recommend you configure it regardless of the value. A general guideline is to use ‘1 day’ or ‘1 week’ depending on the size of each chunk and the general end-user query pattern. For more information,check the documentation on hypertables best practices.Example:select create_hypertable( 'my_table', 'time', chunk_time_interval => INTERVAL '1 day' );Then, send your real-time data to Timescaleandyour existing PostgreSQL production database.Once your new Timescale service has the same schema as your existing production database (but with hypertables), it’s ready to receive the same data. Don’t forget to direct your ingestion processes to Timescale and replicate whatever your method is: Kafka, a client driver, etc.Lastly, fill data into regular tables or non-hypertables.Most Timescale customers have a combination of regular PostgreSQL tables and hypertables. The standard PostgreSQL tables are used as reference tables and can be populated at any time.Backfill As You PleaseAt this point, you should have your real-time production data flowing to two databases:Your existing, soon-to-be legacy PostgreSQL production database, which is linked to your applicationYour new Timescale service, which still isn’t liveAsk yourself: which data do you need to backfill into your new Timescale service to make it operate adequately as your production database? Once you determine that, backfilling can happen at your leisure—you may not even need to backfill data if your retention policy is relatively short.For example, if your retention policy is 30 days, perhaps you prefer to run your existing production database and new Timescale instance in parallel for 30 days instead of backfilling any data. That will give you time for testing, and you won’t need to export/import historical data.✨Editor’s Note: It is worth mentioning that maintaining two databases in parallel for a while will imply extra costs—no benefit comes for free! However, when evaluating the pros and cons of this migration method versus a traditional pg_dump/pg_restore, consider the total associated costs, including the potential downtime effects on your application. That will help you decide which way is best for you.If you must backfill historical data, take note of the earliest data time ingested and use that as the time boundary for the data to be backfilled. This backfilling process is best accomplished by exporting the data in a CSV file and importing it using thetimescaledb_parallel_copy()function.Verify Your DataAnother benefit of using this method is the ability to compare the existing production environment to the new Timescale instance, giving you extra peace of mind on your migration. For example, you can run some key queries on both environments to verify the data is the same.Flip the Switch!Once you feel confident the new data ingested is working as expected, and the necessary historical data has been backfilled, all you have to do is change the connection string on all clients to the new Timescale instance. This is now your new production database! And this migration has been done safely and without downtime.We've Moved. Now What?We hope you learned something new by reading this post—hopefully, you now feel more confident about tackling that PostgreSQL database migration in production that’s been causing you nightmares.And if you’re looking to migrate your PostgreSQL database, have you heard about Timescale? It’s a hosted database platform built on PostgreSQL and TimescaleDB. Now that you know how to migrate quickly, it’s time to explore some of its capabilities: you will get PostgreSQL with extra features for time series (continuous aggregation,compression,automatic retention policies,hyperfunctions). Plus, a platform withautomated backups, high availability, automatic upgrades, flexible resizing with autoscaling, and much more.You can use it for free for 30 days; no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescale-cloud-tips/
2022-11-16T14:00:00.000Z,"Expanding the Boundaries of PostgreSQL: Announcing a Bottomless, Consumption-Based Object Storage Layer Built on Amazon S3","We are excited to announce the initial launch, in private beta, of our new consumption-based, low-cost object storage layer in Timescale. This new capability expands the boundaries of traditional databases, allowing you to transparently tier your data across disk and Amazon S3 while accessing it as if it all lived in one single continuous PostgreSQL table. This means that you can now store an infinite amount of data in Timescale, paying only for what you store. Bottomless cloud storage for time series, events, and analytics is just one piece of our vision to empower you with exceptional data infrastructure so that you can build the next wave of computing.We started Timescale five years ago with a mission: to help developers build the next wave of computing through applications that leverage time-series and real-time analytical data.This mission led us to build TimescaleDB, a time-series database that gives PostgreSQL the performance boost it needs to handle relentless streams of time-series data at scale.On top of scalability and performance, another key concern for developers managing data at scale iscost efficiency. Time-series data is often collected at high frequency or across long time horizons. This scale is often a fundamental part of applications: it’s storing metrics about all IoT devices in a fleet, all the events in a gaming application, or tick data about many financial instruments. But this data adds up over time, often leading to difficult trade-offs about which data to store and for how long.To address this problem, we’ve developed several database features at Timescale aimed at making it easier for developers to manage their time-series data—likenative columnar compression,downsampling,data retention policies, anduser-defined actions. And indeed, these offer massive savings in practice. By compressing data by 95 percent, Timescale ends up much more cost-effective than vanilla storage options like Amazon RDS for PostgreSQL.Today we’re excited to announce how we’re extending this vision to a cloud-native future and building Timescale to supercharge PostgreSQL for time series, events, and analytics at greater scale and lower cost.Timescale now offers consumption-based, low-cost object storage built on Amazon S3.This new storage layer gives you, the developer, more tools to build applications that scale more efficiently while reducing costs. Leveraging a cost-efficient storage layer like Amazon S3 removes the need to pre-allocate—and pay for—an upper bound of your storage. When you tier data on Timescale, you will only pay for what you actually store while retaining the flexibility tokeep a limitless amount of data,and without being charged extra per query.This consumption-based pricing is not only transparent butan order of magnitude cheaperthan our standard disk-based storage. And what’s more, you can access this affordable object storage layer seamlessly from your Timescale database, meaning no need to create a custom pipeline to archive and reload data. All you’ll need is a single SQL command to automatically tier data based on its age, as suited to your application’s needs:# Create a tiering policy for data older than two weeks
SELECT add_tiering_policy ('metrics', INTERVAL '2 weeks');But why stop at cost efficiency? At Timescale, we strive to create aseamless developer experiencefor every feature we release. That means doing the heavy technical lifting under the covers while you continue interacting with your data in the simplest way possible.When applied to cost-saving object storage in Timescale, this means that even when data is tiered, you can continue to query it from within the database via standard SQL, just like you do in TimescaleDB and PostgreSQL. Predicates, filters, JOINs, CTEs, windowing, andhyperfunctionsall work! Reading data directly from tiered object storage only adds a few tens of milliseconds of latency—and this cost goes away for larger scans.We’ve natively architected Timescale databases to support tables (hypertables) that can transparently stretch across multiple storage layers. The object store is thus an integral part of your cloud database rather than just an archive.Here’s an example of theEXPLAINplan for a query that fetches data from disk and object storage (notice theForeign Scan):EXPLAIN SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;


QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
                -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text)The ability to keep your regular and tiered data both accessible via SQL helps you avoid the silos and application-level patchwork that come from operating a separate data warehouse or data lake. It will also help you escape the operational work and extra costs of integrating yet another tool into your data architecture.Starting today, tiering your data to object storageis available for testing in private beta for all Timescale users.Sign up for Timescaleand navigate to the Operations screen, pictured below, to request access. Timescale is free for 30 days, no credit card required.You can request access to our private beta via the Timescale UIBut, this is just the beginning.We plan to further improve object storage in Timescale to not only serve as bottomless, cost-efficient storage but also as ashared storage layerthat makes it dramatically easier for developers to share data across their entire fleet of databases.This is a huge step forward in our vision to build a data infrastructure that extends beyond the boundaries of a traditional database: combining the flexibility of a serverless platform with all the performance, stability, and transparency of PostgreSQL that developers know and love. Not just a managed database in the cloud, but a true “database cloud” to help developers build the next wave of computing.So yes, we are just getting started.✨ A huge “thank you” to the team of Timescale engineers that made this feature possible, with special mention to Gayathri Ayyappan, Sam Gichohi, Vineetha Kamath, and Ildar Musin.To learn more about Timescale’s new data tiering functionality, how it redefines traditional cloud databases, and how it can help you build scalable applications more cost-efficiently, keep reading.Bottomless Storage for PostgreSQLHaving native access to a cloud-native object store means you can now store an infinite amount of data, paying only for what you store. You no longer have to manually archive data to Amazon S3 to save on storage costs, nor import this data into a data warehouse or other tools for historical data analysis. Timescale’s new data tiering feature moves the data transparently to the object store and keeps it available to the Timescale database at all times.To enable this new functionality in PostgreSQL, we built new database internal capabilities and external subsystems. Datachunks(segments of data related by time) that comprise a tiered hypertable now stretch across standard storage and object storage. We also optimized our data format for each layer: block storage starts in uncompressed row-based format and can be converted to Timescale’snative compressed columnar format.On top of that, all object storage is in a compressed columnar format well-suited for Amazon S3 (more specifically,Apache Parquet). This allows developers more options to take advantage of the best data storage type during different stages of their data life cycle.Once a data tiering policy is enabled, chunks stored in our native internal database format are asynchronously migrated into Parquet format and stored in S3 based on their age (although they remain fully accessible throughout the tiering process). A single SQL query will pull data from the disk storage, object storage, or both as needed, but we implemented various query optimizations to limit what needs to be read from S3 to resolve the query.We perform “chunk exclusion” to avoid processing chunks falling outside the query’s time window. Further, the database doesn’t need to read the entire object from S3, even for selected chunks, as it stores various metadata to build a “map” of row groups and columnar offsets within the object. The result? It minimizes the amount of data to be processed, even within a single S3 object that has to be fetched to answer queries properly.Cost-Effective ScalabilityTimescale’s new object storage layer doesn’t just give PostgreSQL bottomless storage but also gives you, the developer, more tools to build applications that scale cost-efficiently.By leveraging Amazon S3, you no longer have to pre-allocate (and pay for) an upper bound of your storage.While Timescale already offers disk auto-scaling, your allocation is still “bumped up” between predefined levels: from 50 GB to 75 GB to 100 GB, from 5 TB to 6 TB to 7 TB, etc. Our new object storage layer scales effortlessly with your data, and you only pay for what you store.These storage savings can be meaningful: an order of magnitude cheaper than employing standard disk-based storage like EBS.So why isn’t this standard for all databases? We build solutions focused on analytical and time-series data. We are doing these transparent optimizations at the larger chunk level rather than the much smaller database page level. This way, we can effectively make the most of S3, which is optimized—for both price and performance—for larger objects. This same approach wouldn’t be practical when employing traditional page-based strategies for database storage.It’s Still Just PostgreSQL, But BetterAs we say,Timescale supercharges PostgreSQL for time series and analytics. But it’s always been important for us to maintain the full PostgreSQL experience, which developers trust and love. This is why we built TimescaleDB as an “extension” of PostgreSQL (although that “extension” has certainly gotten bigger and bigger over the years!).In our books, a smooth developer experience means that developers can continue interacting with all their data as if it’s a standard table—we do the heavy technical lifting under the covers. It should be invisible, and the more our improvements fade into the background, the better.Developers don’t realize that hypertables are actually heavily partitioned data tables—with thousands of such partitions—they just treat them like standard tables. Developers don’t see Timescale’sreal-time aggregationscombining incrementally pre-aggregated data with the latest raw table data to provide them with up-to-date results every time. They are meant to “just work.”We titled our 2017 launch post“When Boring is Awesome: Building a Scalable Time-Series Database on PostgreSQL.”We still strive to make Timescale seem “boring” to developers—simple, fast, scalable, reliable, and cost-effective so that developers can focus their precious time and minds on building applications.This focus on the developer experience similarly motivated our design of transparent data tiering. When data is tiered, you can continue to query tiered data from within the database via standard SQL—predicates and filters, JOINs, CTEs, windowing, andhyperfunctionsall just work.And what’s more, your SQL query will pull relevant data from wherever it is located: disk storage, object storage, or both, as needed, without you having to specify anything in the query.Here’s what it would look like working with relational and time-series data in Timescale, including tiered data. This example shows the use of sensor data, as you might have for IoT, building management, manufacturing, or the like. After creating tables and hypertables forsites,devices, andmetrics, respectively, you use a single commandadd_tiering_policyto set up a policy that automatically tiers data older than two weeks to low-cost object storage.# Create relational metadata tables, including GPS coordinates 
# and FK constraints that place devices at specific sites
CREATE TABLE sites (id integer primary key, name text, location geography(point)); 
CREATE TABLE devices (id integer primary key, site_id integer references sites (id), description text);

# Create a Timescale hypertable
CREATE TABLE metrics (ts timestamp, device_id integer, value float);
SELECT create_hypertable ('metrics', 'ts');

# Create a tiering policy for data older than two weeks
SELECT add_tiering_policy ('metrics', INTERVAL '2 weeks');Now, after you’ve inserted data into your metrics hypertable (and other relational data devices and site information into your relational tables), you can query it as usual. Reading tiered data only adds an extra latency of around tens of milliseconds, and this latency cost may even go away for larger scans.The following SQL query returns the maximum value recorded per device, per day, for a specific site—a fairly standard monitoring use case. Rather than showing the data results (which will just look normal!), we’ll show the output ofEXPLAIN.This command allows developers to see the actual query plan that will be executed by the database, which in this case includes aForeign Scanwhen the database is accessing data from S3. (With our demo data, three chunks remain in standard storage, while five chunks are tiered onto S3.)EXPLAIN SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;

QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
                -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text)Replace Your Siloed Database and Data WarehouseTimescale’s new data tiering functionality expands the boundaries of a traditional cloud database to incorporate features typically attributed to data warehouses or data lakes.The ability to tier data to Amazon S3 within Timescale saves you the manual work of building and integrating up a custom system or operating a separate data store (e.g., Snowflake) for your archival of historical data. Instead of setting up, maintaining, and operating a separate system alongside your production database (and a separate ETL process), you can simply work with a Timescale hypertable that serves your entire data lifecycle, where data is distributed across different storage layers.As we’ve illustrated, you can query regular and tiered data seamlessly from this table and also JOIN it to the rest of your tables, avoiding silos without adding more complexity to your data stack. This not only simplifies operations but also billing: unlike regular data warehousing systems (which typically charge per query, making it very difficult to forecast the final cost), in Timescale you’ll pay only for what you store, keeping your pricing transparent at all times.Request Access to Data Tiering TodayIf you’re already using Timescale,you can test data tiering today by requesting access to our private beta. We welcome your feedback to improve the product and better serve the needs of developers.To start testing data tiering in Timescale today:Sign up to Timescale.The first 30 days are completely free (no credit card required).Log in to the Timescale UI. In your Service screen, navigate to Operations > Data Tiering. Click on the “Request Access” button, and we’ll be in touch soon with the next steps.Bottomless cloud storage for time series, events, and analytics is just one piece of our vision to empower you, the developer, with exceptional data infrastructure so that you can build the next wave of computing.We plan to further improve object storage in Timescale to not only serve as bottomless, cost-efficient storage but also as ashared storage layerthat makes it dramatically easier for developers to share data across their entire fleet of databases.If that sounds interesting to you, please request access to the private beta and let us know!We’re just getting started!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/expanding-the-boundaries-of-postgresql-announcing-a-bottomless-consumption-based-object-storage-layer-built-on-amazon-s3/
2022-09-21T14:13:19.000Z,Timescale Parameters You Should Know About (and Tune) to Maximize Your Performance,"One of the best parts ofbeing a support engineer at Timescaleis working with our customers to get a deep understanding of their use case, making them successful on our cloud service,Timescale. Along the way, we’ve picked up some knowledge on a few key parameters that you should be aware of and tune based on your use case to maximize your performance. In this blog post, we’ll present them to you, sharing our advice on what to look for.Hopefully, this advice will help you make the most of Timescale. Even if you are not a Timescale customer yet, this information may also be useful for you to apply in your own environment—and if you’re curious,you can also try Timescale completely for free. You won’t need a credit card to start the trial, and you'll have access to support the entire time.One caveat to the information below: the optimal setting is typically found via some trial and error. There are too many variables in each user's environment and database configuration for us to be able to write a blog that gives you the optimal settings for all use cases. Yet, we aim to provide some general guidelines that you may find useful.What in The World Are Background Workers?Let’s get started going through our list of key Timescale parameters covering a frequently asked one: background workers.Do you ever see warnings in your log such as...""WARNING: failed to launch job [n] ""Telemetry Reporter [n]"": out of background workers""or,""Compression Policy [n]"": failed to start a background worker?If the answer is yes, you'll find this information useful.Background workersperform background processing for operations specific to TimescaleDB (both live queries and background jobs, all kinds of User-Defined Actions/Policies).The background worker's settings need to be tuned to get the most out of TimescaleDB—issues often arise when worker settings are not properly set. Some of the issues we see often caused by a misconfiguration of background workers are:User-Defined Actions are not working properly.Continuous aggregates are not working properly.Compression policies are not working properly.The retention policies are not working properly.Database size rapidly increases, due to failures in compression and the data retention policies.To avoid these issues, these are three key settings you should look at and our general rule of thumb on how to configure them.timescaledb.max_background_workersYou should configure thetimescaledb.max_background_workerssetting to be equal to the sum of your total number of databases + the total number of concurrent background workers you want running at any given point in time.Note that there is a limit of 1000timescaledb.max_background_workerson Timescale. If you have significantly more than 1000 jobs running simultaneously, we advise you to distribute the execution of your jobs.max_parallel_workersBy default, themax_parallel_workerssetting corresponds to the number of CPUs available.For larger queries, PostgreSQL automatically uses parallel workers if they are available. Increasingmax_parallel_workerscan improve query performance for large queries that trigger the use of parallel workers.max_worker_processesmax_worker_processesdefines the total pool of workers available to both background and parallel workers, as well as a small number of built-in PostgreSQL workers.max_worker_processesshould be AT LEAST 3 (required for checkpointer, WAL writer, and vacuum processes) plus the sum of the background workers and parallel workers:max_worker_processes= 3 +timescaledb.max_background_workers+max_parallel_workers.To learn more abouttimescaledb.max_background_workers,max_parallel_workers, andmax_worker_processes,check out our docs.What to do About Memory Allocation?Now that we've covered workers, let's move on to the next topic: memory allocation.Do You Have Heavy Queries That Aren’t Performing as Expected?Or perhaps you are hitting out-of-memory (OOM) events frequently? If the answer is yes, it may be time to review yourwork_memsetting.work_memdefines the amount of memory to be used by a query operation (such as a sort or hash table) before writing to temporary disk files. Sort operations are used byORDER BY,DISTINCT, and merge joins.The default value for this setting is four megabytes (4MB). If several operations (sort or hash) are running concurrently, then each operation will be allowed to use up towork_memof memory. So, the total memory used could be many times the value ofwork_mem—this situation should be considered when specifying this setting.Ifwork_memis properly tuned, major sort operations are much faster than writing and reading to disk files. If you setwork_memhigher than recommended, it can badly affect the overall available system memory, which can lead to out of memory (OOM) events.To avoid these issues, we recommend settingwork_memas:work_mem= (25% of RAM) / max_connections.However,work_memis assigned whenever a query requests a sort, hash  (or any other structure that requires space allocation), which can occur multiple times per query. It might be safer to assume that max connections * 2 is the amount of memory (RAM) that will be used in practice while devising the recommendedwork_mem. But please, consider that this is dependent on your workload and query patterns.Are Maintenance Operations Taking a Long Time to Execute?maintenance_work_memis another good parameter to look at. It defines the amount of memory to be used by maintenance operations. Maintenance operations includeVACUUM,CREATE INDEX,ALTER TABLE ADD FOREIGN KEYoperations.Settingmaintenance_work_memto less than recommended may lead to a decreased performance of vacuuming and database dump/restore operations. For example, restoring the database usingpg_restorewill usually involve creating some indexes, so it might be worthwhile to increasemaintenance_work_memwhile working withpg_restore.The default value formaintenance_work_memis 64MB. Our recommendation is to set it up to a higher value thanwork_memin order to improve performance forVACUUMoperations and database dump/restore. A good guideline:maintenance_work_mem= 0.05 * total RAM.While configuringmaintanace_work_mem, you should note that memory forshared_buffersandwork_memis already allocated, so you should have enough memory to accommodate: concurrent maintenance operations during peak time *maintenance_work_mem.maintenance_work_memfollows the same guidelines as thework_memsetting, i.e the configuration is per operation, and the necessary memory is allocated on the run. If you are too aggressive with the settings, you may discover out-of-memory errors. It's better to start small, and only increase it once you've done enough monitoring to determine the system's peak memory usage.Are There Memory Considerations if my Workload is Primarily READ Operations vs Primarily WRITE Operations?Another question we hear often.shared_bufferdefines the amount of memory the database server uses for shared memory buffers. The defaultshared_bufferis typically 128 megabytes (128MB), which can be quite low for any typical system.A reasonable amount to start with is 25% of RAM:shared_buffer=25% of RAM.Consider that PostgreSQL requires free memory for system buffers, sort, and maintenance operations, so it is not advisable to setshared_buffersit to a majority of RAM. Nonetheless,shared_buffercan be set to higher than recommended if your use case is mostly READ operations. On the contrary, settingshared_bufferhigher than recommended will decrease performance if your use case is primarily WRITE operations, as all the contents fromshared_buffermust be flushed during write operations.✨Editor's note ✨If you want to big deeper into shared buffers in PostgreSQL,check out our blog post series on the topic.“Out of Shared Memory” Errors?shared_bufferis the biggest chunk of shared memory that PostgreSQL uses, but there might be other components that come into play. One of those components is the Lock Space that’s used to store different types of locks (shared across user processes) by the database.There might be a few reasons you might run out of shared memory, but if your error message in the PostgreSQL logs follows a hint around increasingmax_locks_per_transaction, which is a parameter that controls the average number of object locks allocated for each transaction, this section might interest you.If the error message that appears in the logs looks like this,ERROR:  out of shared memory, you might need to increasemax_locks_per_transaction.The above error can cause queries to fail, as the database can’t grant any more locks. A good check, once you know you’re hitting this, is to see if you’re opening an unbounded number of transactions and never closing them, or maybe they are long-running and touching a lot more objects (eg. tables, indexes, etc). But, if your application really needs a lot more locks, you can tune the parametermax_locks_per_transaction. The default value stands at 64—you might need to raise this value.The recommendation is to choose a number equal to double the maximum number of chunks you can reasonably anticipate having in a hypertable while allowing for some growth. This way we can account for the fact that a hypertable query will typically use as many locks as there are chunks in the hypertable, or twice as many if an index is used.  Thechunks_detailed_sizecommand allows you to view the number of chunks you currently have.A Practical Example: Using This Knowledge To Diagnose And Fix Performance ProblemsTo finish up, we’d like to take a recent example of an issue reported to us (related to the parameters we talked about in this post) and explain how we approached the problem. You can apply the same process to try to fix the performance issues you may be seeing in your own use case.Problem DescriptionDisk usage is increasing rapidly. Unsure what’s causing it.AnalysisChecking the logs, we found frequent mentions of""Compression Policy [N]"": out of background workers.We determined that the jobs related to compression (and also retention policy) were failing, leading to an unexpected increase in disk usage.Relevant SettingsTo review the current background worker settings, we used the following queries:show timescaledb.max_background_workers;show max_worker_processes;show max_parallel_workers;To confirm the current job execution metrics, we used the following queries:select count(*) from timescaledb_information.jobs;select * from timescaledb_information.jobs;select * from timescaledb_information.job_stats;Finally, we wanted to understand the current job distribution pattern, so we used the following query to get a breakdown of each jobsnext_start:select date_trunc('minute', next_start), count(*) from timescaledb_information.jobs group by 1 order by 1,2 desc;AssessmentOn review of the relevant settings,  we found that there were over two thousand user-defined jobs being executed on the database, buttimescaledb.max_background_workerswas set to 8.We also found that the job distribution was extremely narrow,  in that a thousand jobs were set to execute within a 5-minute window when other times of the day had little to no activity scheduled.RecommendationOur first recommendation was to distribute the workload more evenly throughout the day. This can be done by using thenext_startoption:SELECT alter_job(<job_id>, next_start => 'YYYY-MM-DD HH:MI:00.0+00');This allows us to set a reasonable number of background workers to begin with. If jobs continue to fail, then we can increase up to a maximum of 1000 background workers. We decided to start with 100 background workers, and 16 parallel workers, which gave us the following settings:timescaledb.max_background_workers = 100max_worker_processes = 116max_parallel_workers = 16ConclusionIf you’re seeing some unexpected behavior in your Timescale databases, review the parameters listed in this post, and apply the advice we've shared with you. Hopefully, you’ll find it useful!If you still have doubts and would like to get in touch with our support team, it will be our pleasure to work through some of your questions together. If you are a current user of Timescale,you have direct access to support at no extra cost—just shoot us a message at[email protected]. If you’re not yet a Timescale user,you can use it for free for 30 days, no credit card required—and you have full access to our support team during your trial.We look forward to working with you!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescale-parameters-you-should-know-about-and-tune-to-maximize-your-performance/
2019-09-19T19:55:07.000Z,"Analyzing Bitcoin, Ethereum, and 4100+ Other Cryptocurrencies Using PostgreSQL and TimescaleDB","After a Crypto Winter in 2018, cryptocurrencies today are resurging. How can data help us better understand the Crypto Revival?When Satoshi Nakamoto first published the Bitcoin whitepaper in 2008, they probably didn’t foresee the world ofhodlers,lambos,buidlers,bitcoin maximalist carnivores, and n00bs asking “wen moon” in telegram channels, that their actions would create. In 11 years, crypto has gone from something completely esoteric to something seemingly everyone has heard about.2019 has been a big year for crypto, so far. Some of the highlights include: theSEC approving it’s first token sale, theIRS tracking down crypto tax-evaders,university endowments investing in cryptoand evenFacebook announcing it’s own cryptocurrency. We also just this month sawover $1 Billion in Bitcoin transferred in a single transaction. All this, and more, indicates a revived interest in the crypto markets, since the highs of 2017 and lows of 2018, by everyone from institutional investors and banks to lay people trying to side hustle.With the crypto markets once again awash with speculation and hype, it’s important to leverage all the tools at our disposal in order to make sense of the noise. Sometimes reading articles and email newsletters isn’t enough. You have to go directly to the data.As the developers ofTimescaleDB, an open-source time-series database powered by PostgreSQL, we’re data-driven people. So we thought it would be interesting to take adata-driven approachto analyzing the crypto market. For this analysis, we used PostgreSQL and TimescaleDB to analyze market data about Bitcoin, Ethereum and 4196 other cryptocurrencies, and used Tableau to visualize our results.This post shares many high level insights about the crypto market since its inception and during recent years. We answer questions like:How has the price of Bitcoin and Ethereum changed in the past several years?Which new cryptocurrencies have been the most profitable in the past 3 months?What are the cryptocurrencies on the rise?What was the best day to “day-trade” Bitcoin?What countries have the highest trading volume of BTC today?Why is Bitcoin a terrible way to pay for pizza?...and many more, as we dive into analysis of topics like Bitcoin and Ethereum price, new coin growth, trading volume and daily returns.We also share how powerful SQL is as a query language for analyzing time-series data, how TimescaleDB and PostgreSQL further simplify time-series data analysis, and how using the two with Tableau visualizations can surface interesting insights from your data.For the technically curious, you can learn how to create thedatasetwe used for this analysis, load it, and draw insights from it, in thiscompanion tutorial post. In the tutorial, you will findstep by step instructionson how tocreate the dataset using Python(including all code we used for the analysis), how toload the dataintoManaged Service for TimescaleDB, a cloud-hosted version of TimescaleDB, and how to connect your database in the cloud to Tableau torecreate the analysis and produce graphs.About the data used for this analysisFor this analysis, we used historicalOHLCVprice data for over4100 cryptocurrenciesfrom 7/17/2010 to 9/16/2019, courtesy ofCryptoCompareand theirwonderful API. While thedataset we usedonly includes daily data, TimescaleDBeasily scales to handle data from much finer grained time periods.Some of you may recall that we did asimilar analysis on crypto back in 2017, but so much has happened since then, including the addition of almost 3000 more cryptocurrencies and Bitcoin hitting nearly ~$20k in price, that we had to revisit this topic. Where applicable, we’ve included graphs to focus on recent history, from 2017-2019, updating the analysis from our previous post.DISCLAIMER: At Timescale, we help companies harness the power of time series data to make sense of the past, monitor the present, and predict the future. However, nothing in this analysis should be construed as financial advice and we take no liability for your actions as a result of using the information contained in this post. You’re welcome to draw your own conclusions using the tools and data and take your own risks accordingly.So if you’d invested $100 in Bitcoin 9 years ago, today it’d be worth…When analyzing cryptocurrencies, we have to start with the original: Bitcoin. For any beginners, Bitcoincan be thought of as digital gold, because Bitcoin has built-in scarcity (only 21 million BTC will ever be produced), can be almost infinitely divided without losing its unit value, and is difficult to counterfeit. (As an aside, for those looking for an introduction to Bitcoin and other cryptocurrencies, two good places to start areThe Princeton Bitcoin BookandThe Internet of Money.)Looking at historical BTC-USD prices since 2010, we see that BTC prices have slowly increased, with an almost exponential increase taking place between 2014 and 2018.--Query 1
-- BTC 7 day prices
SELECT time_bucket('7 days', time) as period,      
	last(closing_price, time) AS last_closing_price
FROM btc_prices
WHERE currency_code = 'USD'
GROUP BY period
ORDER BY periodFig 1: Bitcoin Closing price in USD from 2010-2019So to answer our original question (and probably create some FOMO), if you bought $100 worth of Bitcoin on 16 September 2010, the price of 1BTC was $0.0619, meaning $100 would have bought you 1615.5088853 BTC. Fast forward 9 years, that $100 would have grown to $16,476,736.67! Queue the lambos! (But of course you would have probably sold when BTC hit $100 back in 29 July 2013 :)).Generating insights like this from time-series data takes no more than a 5 line SQL query thanks to TimescaleDB’s specialtimebucketandlastfunctions, which are special functions exclusively created for TimescaleDB to simplify time-series analysis.Why Bitcoin is a terrible way to pay for pizzaOne thing that’s evident from the data is that Bitcoin is volatile. Bitcoin’s price volatility may mean that its main use case is being a store of value rather than a means of exchange. TheBitcoin Pizza Guy, who paid 10,000 BTC for a 2 pizzas back in 2010, would probably agree, as those 10,000BTC are worth over a $100,000,000 today!Moreover, the time period between 2017 and 2019 saw a lot of ups and downs, so let’s zoom in on that period below:Fig 2: Bitcoin Closing price in USD from July 2017 to July 2019As Figure 2 shows, 2017-2019 has arguably been the most exciting and perhaps also the most painful time in the Bitcoin market, with Bitcoin prices reaching a high of nearly $20,000, before crashing to under $7,000 three months later. That decline continued throughout 2018, with some calling it the first crypto bear market and others a “Crypto Winter”. However, the recent run in price made by Bitcoin since April 2019 may be the first indication of the end of the Crypto Winter, with BTC prices reaching over $12,000 in June 2019.The best day to “day-trade” Bitcoin? February 26, 2014.In order to better understand Bitcoin’s volatility, let’s look at the daily return on BTC, as a factor of the previous day’s rate. This is simple to do usingPostgreSQL window functions, as shown in the query below.--Query 2
-- BTC daily return
SELECT time,
	closing_price / lead(closing_price) over prices AS daily_factor
FROM (  
 SELECT time,         
  closing_price  
FROM btc_prices  
WHERE currency_code = 'USD'  
GROUP BY 1,2
) sub window prices AS (ORDER BY time DESC)Fig 3.1: BTC Daily Return from 2010-2019From Figure 3.1, we can see large amounts of volatility in BTC price from 2010, culminating in a huge spike in daily return factor in February 2014. In fact, within just 7 days we saw the day with the lowest daily return factor — 0.428 on 20 February 2014 — and the highest ever daily return factor — 4.368 on 26 February 2014! It’s no wonder thatstablecoins, or price-stable cryptocurrencies are being looked into as alternatives to use as a means of exchange within apps or for everyday transactions. Imagine paying for everything you buy in Bitcoin and the headaches that daily and weekly BTC price volatility causes.Once again, let’s zoom in on the 2017-2019 time period:Fig 3.2: BTC Daily Return from 2017-2019From Figure 3.2, we see that Bitcoin’s most profitable day since the start of 2017 occurred recently on 25 August 2019, with a daily return of 1.99 times the previous day’s rate. The day with the biggest loss went to 16 January 2018, with a daily return factor of 0.8276, with 14 September 2017 coming in second biggest loss, with a daily return factor of 0.8379.Bitcoin’s top countries by trading volume: US, Japan, South Korea, and... Poland!?Cryptofever has taken the world by storm, with lots of adoption taking place outside of the USA, most notably in places like Europe and Asia. We can get a sense for how crypto is being adopted in different regions of the world by looking at Bitcoin trading volume in different fiat currencies over time.--Query3
-- BTC trading volumes by currency 
SELECT time_bucket('14 days', time) as period,
	currency_code,       
    	sum(volume_btc)
FROM btc_prices
GROUP BY currency_code, period
ORDER BY period;Fig 4: BTC trading volume in different fiat currenciesFrom figure 4 above, we can see that China saw huge amounts of bitcoin trading volume before government intervention which made buying Bitcoin illegal in mid 2017. We can more clearly see how drastic this effect was by looking at Figure 5 below, which is a bar graph version of Figure 4, showing volume of BTC trade in different fiat by year:Fig 5: BTC Trading Volume in different fiat currencies by yearFrom figure 5, we can more clearly see the rise in Chinese (CNY) Bitcoin trading activity and how government intervention in 2017 brought that to a halt. Moreover, we can see Japan (JPY) and South Korea (KRW) overtaking Europe with respect to Bitcoin trading volume, with more volume than the Euro (EUR) during 2017 and 2018. This confirms theUSA, Japan and South Korea as the world’s 3 largest bitcoin markets.Furthermore, if we remove USD, CNY, JPY, KRW and EUR from the list of fiats we can get a sense for the trend in Bitcoin adoption outside the largest markets, as shown in Figure 6 below:Fig 6: BTC Trading Volume in different fiat currencies by year (excluding USD, JPY, KRW, CNY, EUR)Note how South Africa (ZAR) has seen rising BTC trade volumes since 2015, the dramatic increase in trading volume from Hong Kong (HKD) in 2017 and subsequent decrease, and that the currency with the highest trade volume outside the big 5 is none other than the Polish Zloty (PLN)!Figure 6.1 shows trading volumes of BTC in PLN since 2014:Fig 6.1: BTC trading volumes in PLNNow if you’d bought $100 worth of Ethereum in 2015, today it’d be worth...Ethereum is popularly regarded as the cryptocurrency with the second largest interest base after Bitcoin, however, it is fundamentally different from Bitcoin. While Bitcoin is considered to be digital gold, Ether is more like fuel (gas) that runs transactions on the Ethereum network.Since thefirst currency with which you could buy Ethereum was Bitcoin, let’s take a look at historical ETH prices in BTC, shown in Figure 7 below:-- Q4
-- ETH prices in BTC in 7 day intervals
SELECT  
	time_bucket('7 days', time) AS time_period,   
    last(closing_price, time) AS closing_price_btc
FROM crypto_prices
WHERE currency_code='ETH'
GROUP BY time_period
ORDER BY time_periodFig 7: ETH Price in BTC since 2015Figure 7 shows us Ethereum (ETH) closing prices since 3 August 2015 in weekly intervals, expressed in BTC. Notice that 2017 was a rollercoaster year for ETH, with the currency seeing it’s an all time high of 0.1402 BTC on 12 June 2017 and then crashing back down to 0.0288 BTC 4 December 2017, less than 6 months later.Let’s take a look at recent Ethereum prices by zooming in on the period since 2017 in Figure 8 below:Fig 8: ETH Price in BTC since 1 January 2017From Figure 8 above, it seems that ETH then went on another bull run in early 2018, with prices reaching 0.1052 BTC in the period around 22 January 2018. Since then, it seems like ETH prices have been trending downward, with the price on 16 September 2019 reaching 0.0188 BTC. While that’s not great for investors, it may prove to be a blessing for developers and decentralized application users in Ethereum’s ecosystem, as gas costs would be cheaper, perhaps decreasing the barriers to adoption.Crypto Convertibles (not the car kind)Since most people don’t think about prices in BTC (yet), and given how volatile BTC is, it’s useful to also examine ETH prices expressed different fiat currencies. There are two ways this could be done: First by looking at ETH prices directly in different fiat currencies, like we did for for Bitcoin and USD in Figure 1 above. Secondly, we could convert ETH prices in BTC to fiat currency prices, by looking at that day’s BTC to fiat exchange rate. For illustration purposes, we’ll use the second technique. While this may seem like a strange choice, it’s worth noting that conversions from one cryptocurrency to another and then to a fiat currency are fairly common in the cryptocurrency trading world. This is because many exchanges support buying cryptocurrencies with other cryptocurrencies (mainly BTC and ETH), but not all crypto currencies are purchasable directly with fiat currency.In order to examine ETH prices in different fiat currencies in PostgreSQL, we joined two tables and used filters, as the code below illustrates:--Q5
-- ETH prices in fiat
SELECT time_bucket('7 days', c.time) AS time_period,  
	last(c.closing_price, c.time) AS last_closing_price_in_btc,
    last(c.closing_price, c.time) * last(b.closing_price, c.time) FILTER (WHERE b.currency_code = 'USD') AS last_closing_price_in_usd, 
    last(c.closing_price, c.time) * last(b.closing_price, c.time) FILTER (WHERE b.currency_code = 'EUR') AS last_closing_price_in_eur,
    last(c.closing_price, c.time) * last(b.closing_price, c.time) FILTER (WHERE b.currency_code = 'CNY') AS last_closing_price_in_cny,
    last(c.closing_price, c.time) * last(b.closing_price, c.time) FILTER (WHERE b.currency_code = 'JPY') AS last_closing_price_in_jpy,  
    last(c.closing_price, c.time) * last(b.closing_price, c.time) FILTER (WHERE b.currency_code = 'KRW') AS last_closing_price_in_krw
FROM crypto_prices c
JOIN btc_prices b   
	ON time_bucket('1 day', c.time) = time_bucket('1 day', b.time)
WHERE c.currency_code = 'ETH'
GROUP BY time_period
ORDER BY time_periodFig 9.1: ETH Price in USD and EURFig 9.2: ETH Price in JPYFig 9.3: ETH Price in CNYFig 9.4: ETH Price in KRWOne thing to notice is that the Figures 9.1-9.4 all the same shape, since they are another expression of the ETH price in BTC. The main difference in the graphs is the scale of the Y axis, as this reflects the respective currency’s BTC exchange rate. This is as a result of the decision to use the BTC-Fiat exchange rate for this conversion, rather than direct ETH-Fiat prices. When plotted all on the same axis, we get Figure 10 below:Fig 10: ETH Price in Different Fiat CurrenciesFortunately, you can now directly purchase ETH using fiat currency on many exchanges. So let’s look at historical ETH prices in USD,in Figure 11 below:-- ETH prices in USD
SELECT time_bucket('7 days', time) as period,       
	last(closing_price, time) AS last_closing_price
FROM eth_prices
WHERE currency_code = 'USD'
GROUP BY period
ORDER BY periodFig 11: ETH Price in USD from 2015-2019Figure 11 above tells a similar story to that of Figure 8, as we can see the ETH bull run to $1,359 on 8 January 2018. ETH prices in USD have been trending downward since then, with the price on 16 September 2019 falling to $199.So to answer our original question, if you bought $100 worth of Ethereum on 16 September 2015, the price of 1ETH was around $0.89, meaning $100 would have bought you 112.3596 ETH. That $100 would now have grown to $22,359.55! However, the best time to sell would’ve been during the peak Jan 2018, your 112.3596 ETH would’ve been worth $152 696.63! This just goes to show how important it is to time the market!(One piece of analysis which we didn’t do, but encourage readers to do, is to examine the developer activity in the Ethereum ecosystem (eg Github commits/ issues) and see if that correlates to price in some way.)Tracking 4000+ other cryptocurrencies, starting from inceptionWhile we can’t see when exactly coins ICO’d or first got listed on exchanges, wecantrack the date a coin first got added to CryptoCompare as a proxy for its launch date. This allows us to track the launch of different coins over time, as seen in Figure 11.--Q6
--Crypto by date of first data
SELECT ci.currency_code, min(c.time)
FROM currency_info ci JOIN crypto_prices c ON ci.currency_code = c.currency_code
AND c.closing_price > 0
GROUP BY ci.currency_code
ORDER BY min(c.time) DESCFig 11: Number of new cryptocurrencies launched yearIt’s easy to conclude that a bull run in BTC prices might have fueled a massive increase in developer activity in the crypto space. The evidence for this comes from the the 738 new cryptos released in 2017, a year where BTC almost 10x’ed its BTC price between Jan(1 BTC = $2,435 on Jan 1) and December (1 BTC = $19,345 on Dec 16). With bitcoin prices steadily increasing, it’s no wonder that hundreds of developers tried their hand at creating their own cryptocurrency in the hopes of maybe building the next Bitcoin.However, despite the dramatic crash in BTC prices throughout most of 2018, it’s interesting to see the amount of new cryptocurrencies launched in 2018 year being the highest ever, with 771 new cryptocurrency projects launching that year.Figure 12 shows us the result of examining the number of new cryptos launched by day.--Q7
-- Number of new cryptocurrencies by day
SELECT day, COUNT(code)
FROM (  
	SELECT min(c.time) AS day, ci.currency_code AS code  
    FROM currency_info ci JOIN crypto_prices c ON ci.currency_code = c.currency_code  
    AND c.closing_price > 0  
   GROUP BY ci.currency_code  
   ORDER BY min(c.time))a
GROUP BY day
ORDER BY day DESCFig 12: Number of new cryptocurrencies launched by dayTwo days in particular are super interesting. On 2 December 2014, we saw data for 81 cryptocurrencies being added to Cryptocompare and on 26 May 2017, we saw a whopping 134 new cryptos being added on that day!These coins had higher transaction volumes than Bitcoin this month (Hint: It’s not Bitcoin Cash)With over 4000+ cryptocurrencies out there and new ones coming out everyday, it can be hard to pick which ones are worth paying attention to. One helpful metric for spotting coins on the rise is the transaction volume. In Figure 11, we looked at the transaction volume for all 4054 coins in our dataset over the 14 day period from September 2 to 16 2019.--Q8
--Crypto transaction volume during a certain time period
SELECT 'BTC' as currency_code,      
	sum(b.volume_currency) as total_volume_in_usd
FROM btc_prices b
WHERE b.currency_code = 'USD'
AND now() - date(b.time) < INTERVAL '14 day'
GROUP BY b.currency_code
UNION
SELECT c.currency_code as currency_code,      
	sum(c.volume_btc) * avg(b.closing_price) as total_volume_in_usd
FROM crypto_prices c JOIN btc_prices b ON date(c.time) = date(b.time)
WHERE c.volume_btc > 0
AND b.currency_code = 'USD'
AND now() - date(b.time) < INTERVAL '14 day'
AND now() - date(c.time) < INTERVAL '14 day'
GROUP BY c.currency_code
ORDER BY total_volume_in_usd DESCFig 13: Cryptos with the most USD transaction volume over from Sept 2-16 2019It’s surprising to see that Bitcoin, despite being the original cryptocurrency, did not have the largest transaction volume over the time period in question. That honor belonged to USD Tether,USD Tether-- a fiat backed stablecoin, with Ethereum coming in second andLitecoin, the proverbial silver to Bitcoin’s gold, coming in third. Bitcoin (BTC) had the fourth highest USD transaction volume in that 14 day period, followed byRippple(XRP), a global payments system which has partnered with several banks and payment processors, andEOS, a smart contract platform.What are the most profitable new cryptocurrencies?Another way of making sense of the flood of new currencies is to look at how profitable coins are, as measured by total daily return. By honing in on the currencies with the highest increase in rate by day, we can gain a different perspective on which currencies might be worth looking into further.One question to ask is which cryptocurrency has the highest daily return during a certain time period. Figures 14.1 and 14.2 below show cryptocurrencies sorted by their maximum daily return factor between June 16 and September 16 2019.--Q9
--Top crypto by daily return
WITH   
	prev_day_closing AS (
SELECT   
	currency_code,   
    time,   
    closing_price,   
    LEAD(closing_price) OVER (PARTITION BY currency_code ORDER BY TIME DESC) AS prev_day_closing_price
FROM    
	crypto_prices  
),   
	daily_factor AS (
SELECT   
	currency_code,   
	time,   
	CASE WHEN prev_day_closing_price = 0 THEN 0 ELSE closing_price/prev_day_closing_price END AS daily_factor
FROM   
	prev_day_closing
)
SELECT   
	time,   
	LAST(currency_code, daily_factor) as currency_code,   
    MAX(daily_factor) as max_daily_factor
FROM   
	daily_factor
GROUP BY   
	timeFig 14.1: Cryptocurrencies sorted by absolute highest daily returnFrom Figure 14.1 above, we see thatMixin (MIXI), an Ethereum based token, comes out on top with a maximum daily return factor of over 25 million times the previous day’s rate.BOMB, an experimental deflationary currency, had the second highest absolute daily return with a return factor of over 700,000.  We can get a better look at the rest of the data by removing these two outliers, as seen in Figure 14.2 below:Fig 14.2: Cryptos sorted by absolute highest daily return (with MIXI, BOMB removed)From Figure 14.2, we see other cryptos that have had big days in the past 3 months were,Double Eagle Coin (XDE2),Gold Reserve (XGR),SoulCoin (SOUL)andCCCoin (CCC).Furthermore, it’s interesting to note the difference in order of magnitude between the coins with the top daily return in the past 3 months, with MIXI achieving a daily return factor in the millions, BOMB in the hundreds of thousands and the next highest coins in the tens of thousands.Another interesting thing to look at is the amount of times a coin had the top daily return during a certain period of time. Figure 15 shows us the coins with the highest frequency of having the top daily return during the 3 months between 16 June and 16 September 2019.Fig 15: Top coins by most days with highest daily return during a 3 month periodThe coins with the highest frequency of having the top daily return areMIXI(discussed above), with 5 unique days with the top daily return,Bitether (BTR), a cryptocurrency built on the Ethereum platform, with unique 3 days,IceChain (ICHX)coming in third, with two unique days.ConclusionIn this post, we used the power of PostgreSQL and TimescaleDB to analyze a public cryptocurrency dataset of over 4100 cryptocurrencies over the time period 2010 to 2019. We examined time-series trends in Bitcoin and Ethereum prices, new coin growth, trading volume, daily returns, and more.While our analysis aimed to provide a taste of what’s possible using PostgreSQL and TimescaleDB, we encourage you to take the tools we used and apply them to different crypto datasets and gain deeper insights!We’ve created acompanion tutorial postfor those interested in re-creating this analysis or looking for a starting point to perform your own analysis. In the tutorial, you will findstep by step instructionson how to create the dataset using Python (including all code we used for the analysis), how to load the data intoManaged Service for TimescaleDBand how to connect your database in the cloud to Tableau to recreate the analysis and produce graphs. If you do perform your own analysis, let us know what interesting insights you find!Moreover, you can dig in to the technical side of Timescale and how we made PostgreSQL scalable for time-series datain this detailed post. If you’re interested in experiencing the power of Timescale for your time series data, sign up forManaged Service for TimescaleDB!Please drop your comments below and share this post with others whom you think would enjoy it. For follow-up questions or comments, reach out to us on Twitter (@TimescaleDBor@avthars), our communitySlack channel, or reach out to me directly via email (avthar at timescale dot com).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/analyzing-bitcoin-ethereum-and-4100-other-cryptocurrencies-using-postgresql-and-timescaledb/
2019-09-12T16:28:41.000Z,Using AWS Lambda With Managed Service for TimescaleDB for IoT Data,"While it's already quite easy to get started with AWS Lambda and TimescaleDB, here are some tips & tricks to help with your integration.We speak with a ton of users that are capturing IoT data from devices & sensors and storing that data inTimescaleDB. A common pattern we are seeing is IoT data landing in TimescaleDB viaAWS Lambda.The IoT -> Lambda -> TimescaleDB reference architecture looks something like this:For the AWS IoT to AWS Lambda part, AWS provides an IoT Rules Action that can call Lambda functions. You can read more about thathere. So, then how do you go about setting up the AWS Lambda to TimescaleDB integration?In this blog, we want to highlight how easy it is to integrate AWS Lambda with TimescaleDB (which can be easily hosted on Managed Service for TimescaleDB).Let’s get to it!What is AWS Lambda?AWS Lambdais a server-less computing platform provided by Amazon. Lambda makes it very easy to call or automatically trigger function code (that can be written in Go, Node.js, Java or Python) to execute on the platform. You don’t have to worry about server maintenance or scaling… that is all handled by the Lambda service. You just write your code (to effectively do whatever you want), and you have AWS Lambda handle the execution of that code.Using AWS Lambda to access TimescaleDBSince TimescaleDB is powered by PostgreSQL and is accessible via SQL, you can connect to and work with TimescaleDB from just about any code. Since that code can be made into a Lambda function, the integration is really just as simple as:Use a Postgres / SQL client library that is available for your code (in the language of your choice)Write code that speaks SQL to TimescaleDBSetup a Lambda function to execute your codeNotable Tips & TricksEven though the above bullet points are pretty simple, there are a couple tips & tricks we want to highlight to help you along the way.#1 TimescaleDB can be accessed using PostgreSQL client librariesAs mentioned above, TimescaleDB leverages PostgreSQL and is accessible via SQL so you can use client libraries to connect your code to TimescaleDB.Hereis a short listing of client libraries to use with TimescaleDB.#2 Use AWS Lambda Layers to load client librariesClearly, the AWS Lambda folks know your function code will need to pull in libraries and dependencies. To help with this, they include a capability calledAWS Lambda Layerswhich allows you to package (as a ZIP) and upload your dependencies so your function code can import as necessary.For example, if we want to write Lambda function code in Node.js that calls TimescaleDB, we will need the NodePostgreSQL (pg) client libraryavailable for our Node.js code. Use the following instructions (thank you!) to build a ZIP package that includes the pg library that we can load into a Lambda Layer.mkdir node-example
cd node-example
mkdir node-pg-tsdb
cd node-pg-tsdb
npm init
npm install --save pg
cd ..
zip -r node-pg-tsdb.zip node-pg-tsdb -x ""*.DS_Store""Now create the Layer in Lambda by uploading the ZIP file.Add this new Layer to your function code in Lambda.#3 Leverage Lambda environment variables to set client connection propertiesRather than coding your connection parameters directly into your function, you can leverageLambda environment variablesfor these parameters. The environment variables are made available to your code by Lambda during execution.For example, the Node.js pg client library environment settings are documentedhere.You can get the connection setting for your TimescaleDB instance from theManaged Service for TimescaleDBportal. If you are new toManaged Service for TimescaleDB, use thisguideto get started and create your TimescaleDB instance.Now in your function code, just load the library, create a client, and connect. Once those steps are completed, you can leverage SQL and start working with time-series data in TimescaleDB. For reference, you can find the TimescaleDB API documentationhere.const { Client } = require('pg')
const client = new Client();
await client.connect();Advanced Topic: Network SecurityA security best practice is controlling network access to your TimescaleDB instance to the smallest range of IP addresses possible. Managed Service for TimescaleDB supports configuringAllowed IP Addresswhich only allows access if the source traffic is in that specified IP range. To learn how to do this yourself, check out thistutorial.To take advantage of this IP access control when using Lambda, you’ll want your function code to execute & appear as if it’s coming from behind astatic IP. You can use theAllowed IP Addresssetting in Managed Service for TimescaleDB to limit access only from your Lambda function code.For this advanced networking setup in Lambda, I encourage you to take a look at thisblogthat documents how to set up Lambda functions with aNAT GatewayandElastic IPfor a static public IP address. Once the Lambda side is set up, use the static IP and configureAllowed IP Accessfor your TimescaleDB instance from the Timescale Cloud portal to restrict access ONLY to traffic from that IP.What’s NextThe above just showed how easy it is to integrate AWS Lambda with a TimescaleDB instance (that is hosted on Managed Service for TimescaleDB).Here are a couple of resources for you to to try out:Sign up for Timescaleto get a hosted TimescaleDB instance.Try the Timescale MTA example application, where we used Lambda to handle data ingestion via a script on LambdaIngest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/using-aws-lambda-with-timescale-cloud-for-iot-data/
2020-04-22T21:38:27.000Z,"Time-Series Compression Algorithms, Explained","Delta-delta encoding, Simple-8b, XOR-based compression, and more - These algorithms aren't magic, but combined they can save over 90% of storage costs and speed up queries. Here’s how they work.Computing is based on a simple concept:the binary representation of information. And as computing infrastructure has gotten cheaper and more powerful, we have asked it to represent more and more of our information, in the form of data we collect (which is oftentime-series data).But computing is not free. The more efficiently we can represent that information, the more we can save on storage, compute, and bandwidth. Enter compression: “the process of encoding information using fewer bits than the original representation.” (source)Compression has played an important role in computing for several decades. As a concept, compression is even older: “Morse code, invented in 1838, is the earliest instance of data compression in that the most common letters in the English language such as “e” and “t” are given shorter Morse codes.” (source)In this post, we set out to demystify compression.To do this, we explain how several lossless time-series compression algorithms work, and how you can apply them to your own projects.We also explain how we implement them in TimescaleDB, the first open-source relational database to use these time-series compression algorithms, and achieve 90%+ storage efficiencies. [1][1] We use the term “we” throughout this article to represent the engineers who developed this capability:Josh Lockerman,Matvey Arye,Gayathri Ayyapan,Sven Klemm, andDavid Kohn.Why compression matters for time-series dataWe all collecttime-series data. Whether we are measuring our IT systems, web/mobile analytics, product usage, user behavior, sensor/device data, business revenue, etc., time-series data flows through our data pipelines and applications, and enables us to better understand our systems, applications, operations in real-time.One of the challenges of time-series data is storage footprint. In order to analyze data over time, we insert new data (i.e., instead of updating existing data) on every measurement. Some time-series workloads also have high insert rates (e.g., IT monitoring, IoT sensor data). As a result,time-series datasets often scale well into the terabytes and more.In order to achieve high-performance while maintaining resource efficiency, we first identified several best-in-class time-series compression algorithms, and then implemented them in TimescaleDB.These algorithms are quite powerful. According to our users, these algorithms have helped them achieve90%+ lossless compression rates. This translates into 90%+ storage cost savings, which can mean thousands of dollars (and in some cases, tens of thousands of dollars) of savings per year. These algorithms also lead to compute performance improvements: as more data fits in less space, fewer disk pages need to be read to answer queries. [2][2] A 10TB disk volume in the cloud is more than $12,000 per year itself (at $0.10/GB/month for AWS EBS storage), and additional HA replicas and backups can grow this number by another 2-3x. Achieving 95% storage can save you over $10K-$25K per year in storage costs alone ($12K/10TB * 10TB/machine * 2 machines [one master and one replica] * 95% savings = $22.8K).What are these magical time-series compression algorithms?First of all, they’re not magic, but clever computer science techniques. Here are the set of compression algorithms we'll explain, grouped by data type:Integer compression:Delta encodingDelta-of-delta encodingSimple-8bRun-length encodingFloating point compression:XOR-based compressionData-agnostic compression:Dictionary compressionInteger compressionDelta-encodingDelta-encoding(also known as Delta compression) reduces the amount of information required to represent a data object, by only storing the difference (ordelta) between that object and one or more reference objects. These algorithms work best where there is a lot of redundant information: for example, in versioned file systems (e.g.,this is how Dropbox efficiently syncs your files).Applying delta-encoding to time-series data makes a lot of sense: we can use fewer bytes to represent a data point by only storing the delta from the previous data point. (In fact, given enough coffee and time, we would argue that versioned file systems themselves are time-series datasets, but we’ll save that discussion for another time.)For example, imagine we were collecting a dataset that collected CPU, free memory, temperature, and humidity over time (time stored as an integer value, e.g., # seconds since UNIX epoch).Under a naive approach, we would store each data point with its raw values:| time                | cpu | mem_free_bytes | temperature | humidity |
|---------------------|-----|----------------|-------------|----------|
| 2020-04-01 10:00:00 | 82  | 1,073,741,824  | 80          | 25       |
| 2020-04-01 10:05:00 | 98  | 858,993,459    | 81          | 25       |
| 2020-04-01 10:05:00 | 98  | 858,904,583    | 81          | 25       |With delta-encoding, we would only store how much each value changed from the previous data point, resulting in smaller values to store:| time                | cpu | mem_free_bytes | temperature | humidity |
|---------------------|-----|----------------|-------------|----------|
| 2020-04-01 10:00:00 | 82  | 1,073,741,824  | 80          | 25       |
| 5 seconds           | 16  | -214,748,365   | 1           | 0        |
| 5 seconds           | 0   | -88,876        | 0           | 0        |Now, after the first row, we are able to represent subsequent rows with less information.Applying Delta-encoding to time-series data takes advantage of the fact that most time-series datasets are not random, but instead represent something that is slowly changing over time.The storage savings over millions of rows can be pretty substantial, especially when the value doesn’t change at all.Additional reading:Delta Compression TechniquesDelta-of-delta encodingDelta-of-delta encoding (also known as “delta-delta encoding”), takes delta encoding one step further: it applies delta-encoding a second time over delta-encoded data. With time-series datasets where data collection happens at regular intervals, we can apply delta-of-delta encoding to thetimecolumn, effectively only needing to store a series of 0’s.Applied to our example dataset, we now get this:| time                | cpu | mem_free_bytes | temperature | humidity |
|---------------------|-----|----------------|-------------|----------|
| 2020-04-01 10:00:00 | 82  | 1,073,741,824  | 80          | 25       |
| 5 seconds           | 16  | -214,748,365   | 1           | 0        |
| 0                   | 0   | -88,876        | 0           | 0        |In this example, delta-of-delta further compresses “5 seconds” down to “0” for every entry in thetimecolumn after the second row. (Note that we need two entries in our table before we can calculate the delta-delta, because we need two delta’s to compare.)This compresses a full timestamp (8 bytes = 64 bits) down to just a single bit (64x compression).(In practice we can do even better by also applying Simple-8b + RLE. More below.)In other words, delta-encoding stores the first derivative of the dataset, while delta-of-delta encoding stores the second derivative of the dataset.Simple-8bWith Delta (and delta-of-delta) encoding, we’ve reduced the number of digits we needed to store. Yet we still need an efficient way to store these smaller integers. Here’s an example that illustrates why: Say, in our previous example, we still use a standard integer datatype (which takes 64 bits on a 64-bit computer) to represent the value of “0” when delta-delta encoded. Thus, even though we are storing “0”, we are still taking up 64 bits  – we haven’t actually saved anything.Enter Simple-8b, one of the simplest and smallest methods of storing variable length integers.In Simple-8b, the set of integers is stored as a series of fixed-size blocks. For each block of integers, each integer is represented in the minimal bit-length needed to represent the largest integer in that block. The first bits of each block denotes that minimum bit-length for the block.This technique has the advantage of only needing to store the length once for a given block, instead of once for each number. Also, since the blocks are of a fixed-size, we can infer the number of integers in each block from the size of the integers being stored.As an example, say we were storing the changing temperature over time, applied delta encoding, and ended up needing to store this set of numbers:| temperature (deltas) |
|----------------------|
| 1                    |
| 10                   |
| 11                   |
| 13                   |
| 9                    |
| 100                  |
| 22                   |
| 11In other words, our data looks like this:1, 10, 11, 13, 9, 100, 22, 11With a block size of 10 digits, we could store this set of numbers in a Simple-8b-like scheme as two blocks, one storing 5 2-digit numbers, and a second storing 3 3-digit numbers.{2: [01, 10, 11, 13, 09]} {3: [100, 022, 011]}As you can see, both blocks store about 10-digits worth of data, even though some of the numbers have to be padded with a leading ‘0’. (Note in our example, the second block only stores 9 digits, because 10 is not evenly divisible by 3).Simple-8b works very similarly, except it uses binary numbers instead of decimal ones, and generally uses 64-bit blocks. In general, the larger number length, the fewer number of numbers that can be stored in each block:Source. A selector value of “2” corresponds to a number length of 1, “3” corresponds to a number length of 2, and so on. Please note: The version of Simple-8b implemented in TimescaleDB has slightly different sizes than this version, in order to handle 64-bit values and RLE. More below.Additional reading:Decoding billions of integers per second through vectorizationincludes an even more detailed description of how Simple-8b works.Run-length encoding (RLE)Simple-8b compresses numbers very well, using approximately the minimal number of digits for each number.However, in certain cases we can do even better.We can do better in cases where we end up with a large number of repeats of the same value. This can happen if the value does not change very often, or (if you recall from the delta and delta-delta section) if an earlier transformation removes the changes.As one example, consider our delta transformation of the time and humidity values from earlier. Here, the time column value repeats with “5”, and the humidity column with “0”:| time                | cpu | mem_free_bytes | temperature | humidity |
|---------------------|-----|----------------|-------------|----------|
| 2020-04-01 10:00:00 | 82  | 1,073,741,824  | 80          | 25       |
| 5 seconds           | 16  | -214,748,365   | 1           | 0        |
| 5 seconds           | 0   | -88,876        | 0           | 0        |To see how we can do better in representing these repeats of the same value, let’s actually use a less simplified example. Say we were still storing the changing temperature over time, we again applied delta-encoding, but now ended up with this set of numbers:| temperature (deltas) |
|----------------------|
| 11                   |
| 12                   |
| 12                   |
| 12                   |
| 12                   |
| 12                   |
| 12                   |
| 1                    |
| 12                   |
| 12                   |
| 12                   |
| 12                   |In other words, our data now looks like this:11, 12, 12, 12, 12, 12, 12, 1, 12, 12, 12, 12For values such as these, we do not need to store each instance of the value, but merely how long therun, or number of repeats, is. We could store this set of numbers as{run; value}pairs like this:{1; 11}, {6; 12}, {1; 1}, {4; 12}This technique only takes 11 digits of storage ([1, 1, 1, 6, 1, 2, 1, 1, 4, 1, 2]), as opposed to the approximately 23 digits that an optimal series of variable-length integers would require ([11, 12, 12, 12, 12, 12, 12, 1, 12, 12, 12, 12]).This is Run-length encoding (RLE), which is one of the classic compression algorithms (along with Dictionary compression, discussed later). If you see compression with seemingly absurd ratios -- e.g., fewer than 1 bit per value -- run-length-encoding (or a similar technique) is probably being used. Think about a time-series with a billion contiguous 0’s, or even a document with a million identically repeated strings: both run-length-encode quite well.RLE is used as a building block in many more advanced algorithms: e.g.,Simple-8b RLE, an algorithm that combines both techniques.In practice, in TimescaleDB we implement a variant of Simple-8b RLE, where we detect runs on-the-fly, and run-length-encode if it would be beneficial. In this variant, we use different sizes than standard Simple-8b, in order to handle 64-bit values, and RLE. More information (and the source code) for the variant we built into TimescaleDB can be foundhere.Floating point compressionXOR-based compressionGorilla, an in-memory time-series database developed at Facebook (and research paper published in 2015), introduced two compression techniques that improve on delta-encoding. The first is delta-of-delta encoding for timestamps, which we already covered above. Here we cover the second,XOR-based compression, which is something that typically applies to floats.(Note: Developers will often refer to “Gorilla compression”, which is generally at least one, if not both, of these techniques.)Floating point numbers are generally more difficult to compress than integers. Unlike fixed-length integers which often have a fair number of leading 0s, floating point numbers often use all of their available bits, especially if they are converted from decimal numbers, which can’t be represented precisely in binary. [3][3] Decimal is in base-10, while floats are in base-2 (binary). This means that decimal numbers can be built out of a series of base-10 fractions, like a/10 + b/100 + c/1000 ... etc, while floats are built out of base-2 fractions like a/2 + b/4 + c/8 ... etc. The numbers representable as sums of these different fractions don't completely overlap, so decimal numbers are rounded to the nearest value in binary. This causes some numbers that are simple to represent in base-10, like 93.9, to get represented in a float as much more complicated numbers, which gets represented as approximately 93.90000152587890625 in binary.Furthermore, techniques like delta-encoding don’t work well for floats, as they do not reduce the number of bits sufficiently. For example, in our example above, if we stored CPU as a double, the delta remains a number with many significant digits:| time  | cpu          |
|-------|--------------|
| t1    | 0.8204859587 |
| t2    | 0.9813528043 |
| DELTA | 0.1608668456 |Due to these challenges, most floating-point compression algorithms tend to be either complex and slow, or lossy (e.g., by truncating significant digits). (It’s important when evaluating compression algorithms to distinguish between lossless and lossy compression: for example, in the above example, if we truncate the cpu float values to two significant digits, the delta of 0.16 is already much smaller, but we have lost information.)One of the few simple and fast lossless floating-point compression algorithms is the XOR-based one that the Gorilla paper applies, with very good results:“We addressed this by repurposing an existing XOR based floating point compression scheme to work in a streaming manner that allows us to compress time series to an average of 1.37 bytes per point, a 12x reduction in size.”(source)In this algorithm, successive floating point numbers are XORed together, which means that only the different bits are stored. (Quick primer on how a binary XOR operation works.)Source: Gorilla paperThe first data point is stored with no compression. Subsequent data points are represented using their XOR’ed values, encoded using a bit packing scheme covered in detail in thepaper(and also neatly diagrammed in thisblog post).According to Facebook, with this compression algorithm, over 50% of floating point values (all doubles) were compressed to a single bit, ~30% to 26.6 bits, and the remainder to 39.6 bits. (Reminder, a double is 64 bits).Source: Gorilla paperAdditional reading:Gorilla: A Fast, Scalable, In-Memory Time Series DatabaseData-agnostic compressionDictionary compressionOne of the earliest lossless compression algorithms,Dictionary compression(in particular,LZ-based compression) is the ancestor of many compression schemes used today, includingLZW(used inGIF) andDEFLATE(used inPNG,gzip).(As a general concept, dictionary compression can also be found in areas outside of computer science: e.g., in the field ofmedical coding.)Instead of storing values directly, Dictionary compression works by making a list of the possible values that can appear, and then just storing an index into adictionarycontaining the unique values. This technique is quite versatile, can be used regardless of data type, and works especially well when we have a limited set of values that repeat frequently.For example, say we had an additional column in our time-series dataset storing city location for each measurement:| City          |
|---------------|
| New York      |
| San Francisco |
| San Francisco |
| Los Angeles   |
| ⋮             |Instead of storing all the City names directly, we could instead store a dictionary, such as{0: “New York”, 1: “San Francisco”, 2: “Los Angeles”, ...}and just store the indices[0, 1, 1, 2, ...]in the column.For a dataset with a lot of repetition, like the one above, this can offer significant savings. In the above dataset, each city name is on average 11 bytes in length, while the indices are never going to be more than 4 bytes long, reducing space usage nearly 3x.(In TimescaleDB, we compress the list of indices even further using the Simple8b+RLE scheme described earlier, making the storage cost even smaller.)Like all compression schemes, Dictionary compression isn’t always a win: in a dataset with very few repeated values, the dictionary will be the same size as the original data, making the list of indices into the dictionary pure overhead.However, this can be managed: in TimescaleDB, we detect this case, and then fall back to not using a dictionary in those scenarios.Compression in practiceTimescaleDB is an open-source time-series database, engineered on PostgreSQL, that employs all of these best-in-class compression algorithms to enable much greater storage efficiency for our users (over 90% efficiency, as mentioned earlier).TimescaleDB deploys different compression algorithms, depending on the data type:Delta-of-delta + Simple-8b with run-length encoding compressionfor integers, timestamps, and other integer-like typesXOR-based compressionfor floatsWhole-row dictionary compressionfor columns with a few repeating values (plus LZ compression on top)LZ-based array compressionfor all other typesIn particular, we extended classic XOR-based compression and Simple-8b so that we could decompress data in reverse order. This enables us to speed up queries that use backwards scans, which are common in time-series query workloads. (For super technical details, please see ourcompression PR.)We have found this type-specific compression quite powerful: In addition to higher compressibility, some of the techniques like XOR-based compression and Delta-of-delta can be up to 40x faster than LZ-based compression during decoding, leading to faster queries.In addition, if we have data for which none of the aforementioned schemes work, we store the data directly in a specialized array. Doing this provides two notable benefits.For one, we can compress both the nulls-bitmap of the array, and the sizes of the individual elements using our integer-compression schemes, getting some small size savings.Another benefit is that this allows us to convert many (e.g., 1000) database rows into a single row. PostgreSQL stores a non-negligible overhead for each row stored (around 48 bytes last we checked), so for narrow rows, removing that overhead can be a non-trivial savings.For even more informationFor more information on how we built this capability into TimescaleDB, please readthis articleandthis section of our docs.If you’d like to test out TimescaleDB and see compression in action,you can get startedhere. And if you have any further questions, pleasejoin our community onSlack.Suggested additional reading:History of Lossless Data Compression AlgorithmsWhat the heck is time-series data?Delta Compression Techniques[2018]Decoding billions of integers per second through vectorization[2012] (includes a detailed description of how Simple-8b works)Gorilla: A Fast, Scalable, In-Memory Time Series Database[2015]Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/time-series-compression-algorithms-explained/
2020-03-23T16:50:26.000Z,"Homegrown Monitoring for Personal Projects: Prometheus, TimescaleDB, and Grafana FTW","Learn how Tyler - one of our newest additions to the team - setup a ""roll-your-own monitoring"" solution with 99% storage savings – and how you can do the same, be it for your personal projects or a critical piece ofbusinessinfrastructure.Like many of you, I love to tinker. I overcomplicate the things in and around my home because, while I don’tNEEDan enterprise-ready network, it’s fun to play with anyway. I’ve got anOwncloudinstance I run for a friend, api-hole, a lab Postgres instance, a CIFS server for backing up thedashcam and sentry footage from my Tesla, and a Docker host for running various miscellany (including all the bits for this project!), plus other assorted things.As a new addition to the Timescale team, I decided I needed a good dogfood project, and monitoring all of these things felt like a good way to get started. So, I did some reading and decided to set upPrometheus,TimescaleDB(+ theTimescale Prometheus Adapter), andGrafanato get acquainted with the various pieces.I hoped to get a better sense of how our customers might be using our software, as well as just get more familiar with TimescaleDB, Postgres, Grafana – and how they all fit together. Plus, it’d be good to know if my Owncloud server was running out of storage, or if that pi-hole was running out of RAM.My SetupPrometheus was straight-forward for me to get up and going; their focus on discrete parts doing one thing and doing them well really shines.The Prometheus server and the exporters are both hassle-free: run the exporter on the host you want to monitor, point the prometheus server at the right host and port, and you're done!The node-exporter is lightweight, and the settings for Prometheus itself are likewise minimal, making for a quick and easy solution.If you’d like to tinker along with me, I’ve got an example of what I’m using for the monitoring bitson GitHub(PRs always welcome!). Did I mention I like to overcomplicate things? :)In any event, I’m gathering Prometheus node-exporter data from 4 hosts, and Prometheus pi-hole-exporter data from the pi-hole in addition.Here’s a look at what this looks like, with different parts at two locations(some at my office, and some at home).Quick sketch of my architecture setupThis monitoring setup generates about 3.5GB of raw data every day - not a lot in terms of monitoring for a business, but quite a lot for a side project! Since this is just for some messing around, I don’t want to have to put a bunch of disk storage behind it.Here's the monitoring view, usinga great dashboardby starsL.Node-Exporter Monitoring DashboardWhat to do with my data?I considered dropping data older than a few days, but before I did that, I thought I should check outTimescaleDB’s native compression.My reasoning: the more data you can collect from your systems, the better you can see trends, plan for growth, and understand ideal service windows for maintenance. Not super important for home stuff, but in the interest of making this as practical as possible, I wanted the full picture of the insights I could glean. To do that, I needed to grab and retain as much data as I reasonably could.Enabling compression? Just as straightforward as getting Prometheus up and running 🎉:ALTER TABLE metrics_values 
SET (timescaledb.compress, timescaledb.compress_orderby = 'labels_id, time DESC');Then, I set my policy to compress data older than 3 days:SELECT add_compress_chunks_policy
('metrics_values', INTERVAL '3 days');I’ve let this run for a couple weeks now, and the results have been astounding. I figured that I’d be able to keep roughly two weeks worth of data before rolling it off – but the compression on my Prometheus data is so effective,I’ll be able to keep months worth of data without using any significant storage(1.5 GB/month rather than about 105GB/month). This lets me see much longer usage patterns.See for yourself:SELECT uncompressed_total_bytes, compressed_total_bytes 
FROM timescaledb_information.compressed_hypertable_stats;uncompressed_total_bytescompressed_total_bytes58 GB637 MBAdmittedly, this is just a single sample with a relatively small amount of data, but I’m seeing compression coming in at roughly 99% space savings (approximately 58000MB →  637MB) for various kinds of Prometheus data (no small feat!).If you want more information about how compression works and the benchmarks we saw during internal testing, check out thisblog post.If you want to try it out yourself and see how much your data compresses,our Quick Start documentationis a great place to start.If you have questions, need help troubleshooting, or want to get some recommendations,reach out to us on Slack at any time.There’s always at least one Timescale Engineer - including myself - “on duty” and community members are keen to jump in and help too.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/homegrown-monitoring-for-personal-projects-prometheus-timescaledb-and-grafana-ftw/
2020-03-05T19:11:10.000Z,Charting the Spread of COVID-19 Using Data,"Data gives us insight into the world around us – and allows us to visualize and prepare for the spread of the COVID-19 pandemic.In recent months, we’ve seen thespread of the COVID-19 virus(commonly referred to as ""coronavirus""). Not only are we a globally distributed team, some Timescale team family members see fears and concerns first-hand while working in our emergency rooms.First and foremost, our thoughts are with those who've been affected in any way. We join the global community in hoping for the best in the coming weeks and months, especially in containing transmission and reducing the virus' spread.We were recently made aware of atime-series datasetfrom Johns Hopkins University, containing daily case reports from around the world. Timescale community memberJoel Natividadcreated thisrepo of time-series utilitiesthat enables people to use the Johns Hopkins University dataset with TimescaleDB to better monitor the COVID-19 situation.In this post, we'll walkthrough how to load the dataset into TimescaleDB and use Grafana to visualize queries. Our hope is that you’ll be able to use this dataset and tutorial as a starting point for your own analysis (extending upon the queries we include), or to share information with colleagues, family, and friends.You can getTimescaleDBin two forms: Managed Service for TimescaleDB (with free credits) or TimescaleDB for download. You canfollow these instructionsto choose your installation method. If you haven’t done so, you’ll also want toinstall psqlto access the database from your command line.Ingesting the datasetOnce you’ve installed TimescaleDB, you will need to load the dataset. First, you will want to clone the GitHub repository Joel created:git clone https://github.com/dathere/covid19-time-series-utilities.gitThis repository contains several files, which are explained in theREADME. Setup your database and ingest the dataaccording to the instructionsin the GitHub repository.Once the data is fully ingested, you should be able to login to your database frompsql:psql -x ""postgres://[YOUR USERNAME]:[YOUR PASSWORD]@[YOUR HOST]:[YOUR PORT]/[DB_NAME]?sslmode=require""Once you’re logged in, you will be able to run the following query:SELECT * FROM covid19_ts LIMIT 5;Your output should look something like this:province_state | country_region |    observation_date    | confirmed | deaths | recovered
---------------+----------------+------------------------+-----------+--------+-----------
Hubei          | Mainland China | 2020-01-31 23:59:00+00 |      5806 |    204 |       141
Zhejiang       | Mainland China | 2020-01-31 23:59:00+00 |       538 | [null] |        14
Guangdong      | Mainland China | 2020-01-31 23:59:00+00 |       436 | [null] |        11
Henan          | Mainland China | 2020-01-31 23:59:00+00 |       352 |      2 |         3
Hunan          | Mainland China | 2020-01-31 23:59:00+00 |       332 | [null] |         2
(5 rows)At this point, your time-series database is properly configured and you've loaded the COVID-19 dataset.Note: the Johns Hopkins University data is a running total, not a per-day total. So, the data for February 23rd for any given location represents a cumulative tally of all cases in that location as of February 23rd.Before we continue, we need to clean up our dataset.In some cases, a given row in thecovid19_tsorcovid19_locationstable may not have aprovince_stateentered properly. Cleaning public datasets is fairly common, and fortunately, this dataset is easy to prepare for the rest of our tutorial. Inpsqlenter the following commands:UPDATE covid19_locations 
SET province_state = 
    CASE WHEN province_state = '' 
    THEN country_region 
    ELSE province_state 
    END;At this point, all our rows have either a specific Province or State in theprovince_statecolumn, or, failing that, we default to thecountry_region.Now, let’s start running queries that will give us some insight into the pandemic.How many confirmed cases were reported worldwide each day?Let’s start by identifying how many cumulative cases have been identified each day; the dataset starts on January 22nd, so that’s where we'll start for our query.SELECT time_bucket('1 day', observation_date) AS day,
       SUM(confirmed)
FROM covid19_ts
GROUP BY day
ORDER BY day;Your output should look something like this:day           |  sum
-----------------------+-------
2020-01-22 00:00:00+00 |   555
2020-01-23 00:00:00+00 |   653
2020-01-24 00:00:00+00 |   941
2020-01-25 00:00:00+00 |  1438
2020-01-26 00:00:00+00 |  2118
2020-01-27 00:00:00+00 |  2927
2020-01-28 00:00:00+00 |  5578
2020-01-29 00:00:00+00 |  6165
2020-01-30 00:00:00+00 |  8235
2020-01-31 00:00:00+00 | 10037
2020-02-01 00:00:00+00 | 12009
2020-02-02 00:00:00+00 | 16685
2020-02-03 00:00:00+00 | 19719
2020-02-04 00:00:00+00 | 23821
2020-02-05 00:00:00+00 | 27457
(edited for length)How many confirmed cases have been reported each day near Milan, Italy?As of the publication date of this post, northern Italy has the highest concentration of COVID-19 cases, outside of China, and it continues to intensify.What if we wanted to see the growth rate of COVID-19 in Italy?To do this, we query on thecountry_regionfield in our database:SELECT * FROM covid19_ts WHERE country_region = 'Italy';Our output will look something like this:province_state | country_region | observation_date       | confirmed | deaths | recovered
---------------+----------------+------------------------+-----------+--------+-----------
               | Italy          | 2020-01-31 23:59:00+00 |         2 | [null] |    [null]
               | Italy          | 2020-01-31 08:15:00+00 |         2 |      0 |         0
               | Italy          | 2020-01-31 08:15:53+00 |         2 |      0 |         0
               | Italy          | 2020-02-07 17:53:02+00 |         3 |      0 |         0
               | Italy          | 2020-02-21 23:33:06+00 |        20 |      1 |         0
               | Italy          | 2020-02-22 23:43:02+00 |        62 |      2 |         1
               | Italy          | 2020-02-23 23:43:02+00 |       155 |      3 |         2
               | Italy          | 2020-02-24 23:43:01+00 |       229 |      7 |         1
               | Italy          | 2020-02-25 18:55:32+00 |       322 |     10 |         1
               | Italy          | 2020-02-26 23:43:03+00 |       453 |     12 |         3
               | Italy          | 2020-02-27 23:23:02+00 |       655 |     17 |        45
               | Italy          | 2020-02-28 20:13:09+00 |       888 |     21 |        46
               | Italy          | 2020-02-29 18:03:05+00 |      1128 |     29 |        46
               | Italy          | 2020-03-01 23:23:02+00 |      1694 |     34 |        83
               | Italy          | 2020-03-02 20:23:16+00 |      2036 |     52 |       149
(15 rows)How many confirmed cases have been reported each week near me?What if we wanted to look at cases near a specific location?This requires that we make use of thelatitudeandlongitudeinformation stored in our database. To use the location, we’ll need to get our tables ready for geospatial queries.One caveat: The Johns Hopkins University dataset has limited information about specific locations. All cases in Italy are marked as “Italy,” for example, while cases in China and the United States may be more specific with regional or city locations.We can use the PostGIS PostgreSQL extension to help analyze geospatial data. (This is available out-of-the-box onManaged Service forTimescaleDB, or you canmanually installfor self-hosted versions.) PostGIS allows us to slice data by time and location within TimescaleDB.CREATE EXTENSION postgis;Then, run the\dxcommand inpsqlto verify that PostGIS was installed properly. You should see the PostGIS extension in your extension list, as noted below:List of installed extensions
Name        | Version |   Schema   |                             Description
------------+---------+------------+---------------------------------------------------------------------
plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
postgis     | 2.5.3   | public     | PostGIS geometry, geography, and raster spatial types and functions
timescaledb | 1.6.0   | public     | Enables scalable inserts and complex queries for time-series data
(3 rows)Thecovid19_locationstable in our database includes the locations in the Johns Hopkins University dataset, as well as their latitude and longitude coordinates.For example, we can see all the locations in the United States like so:SELECT * FROM covid19_locations WHERE country_region = 'US' LIMIT 10;The output for this query looks something like this:province_state                | country_region | latitude | longitude 
---------------------------------------------+----------------+----------+-----------
 Unassigned Location (From Diamond Princess) | US             |  35.4437 |  139.6380
 King County, WA                             | US             |  47.5480 | -121.9836
 Santa Clara, CA                             | US             |  37.3541 | -121.9552
 Snohomish County, WA                        | US             |  48.0330 | -121.8339
 Cook County, IL                             | US             |  41.7377 |  -87.6976
 Fulton County, GA                           | US             |  33.8034 |  -84.3963
 Grafton County, NH                          | US             |  43.9088 |  -71.8260
 Hillsborough, FL                            | US             |  27.9904 |  -82.3018
 Providence, RI                              | US             |  41.8240 |  -71.4128
 Sacramento County, CA                       | US             |  38.4747 | -121.3542
(10 rows)We need to alter thecovid19_locationstable to work with PostGIS.To start, we’ll add geometry columns for case locations:ALTER TABLE covid19_locations
    ADD COLUMN location_geom geometry(POINT, 2163);Next we’ll need to convert the latitude and longitude points into geometry coordinates (so they play well with PostGIS).UPDATE covid19_locations
    SET location_geom = ST_Transform(ST_SetSRID(ST_MakePoint(longitude, latitude), 4326), 2163);Lastly, let’s say we want to examine cases near Seattle, in the United States. Seattle is located at (lat, long) (47.6062,-122.3321).Our database is setup, so we're ready to run a geospatial query that returns the number of confirmed COVID-19 cases within 75km of Seattle, each day, since the start of our dataset. (TimescaleDB is based on PostgreSQL, so we can use all our favorite SQL functions.)In this case, we need to use a subquery for the information in ourcovid19_locationstable and a broader query for all confirmed cases that match the locations in our subquery.First, we write the subquery by querying thecovid19_locationstable for all locations within 75km of Seattle:SELECT *
FROM covid19_locations
WHERE ST_Distance(location_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-122.3321, 47.6062), 4326), 2163)) < 75000;We should get output that looks like this:province_state       | country_region | latitude | longitude |                   location_geom
---------------------+----------------+----------+-----------+----------------------------------------------------
King County, WA      | US             |  47.5480 | -121.9836 | 010100002073080000A2B8C609FEC838C1004A6D25430F1F41
Snohomish County, WA | US             |  48.0330 | -121.8339 | 010100002073080000344A7104C26438C1768752D481082141
(2 rows)Now, we need to find all confirmed cases in ourcovid19_tsdataset whoseprovince_statematches the result of our first query.SELECT time_bucket('1 day', observation_date) as day,
       SUM(confirmed) AS near_sea
FROM covid19_ts
WHERE province_state IN (
    SELECT province_state
    FROM covid19_locations
    WHERE ST_Distance(location_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-122.3321, 47.6062), 4326), 2163)) < 75000
)
GROUP BY day
ORDER BY day;Your output should look something like this:day           | near_sea 
------------------------+----------
 2020-02-29 00:00:00+00 |        1
 2020-03-01 00:00:00+00 |        2
 2020-03-02 00:00:00+00 |       18
 2020-03-03 00:00:00+00 |       27
 2020-03-04 00:00:00+00 |       39
(5 rows)Use Grafana to visualize confirmed cases by geographic locationWe have the dataset, we have our queries, and, now, let's take the COVID-19 dataset and visualize the total number of confirmed cases by geographic location.To create our visualization, we’ll use Grafana. Follow ourshort tutorial on getting started with Grafanato setup Grafana in Managed Service for TimescaleDB, your desktop, or using Grafana Cloud, using the dataset you’ve been using in this post.In Grafana, create a new dashboard and add a new visualization. In the visualization menu, search for and select the WorldMap panel type.To build our visualization, we will need our information organized properly in our query:For each dayA sum of all confirmed casesGrouped by theprovince_statewhere they happenedPlotted on the map using the latitude and longitude for theprovince_stateOrdered by timeTo build this query, we use a SQLINNER JOINstatement to combine information from thecovid19_tsdataset and thecovid19_locationstables:SELECT time_bucket('1 day', covid19_ts.observation_date) AS day,
       SUM(covid19_ts.confirmed) AS confirmed, 
       covid19_ts.province_state as location, 
       covid19_locations.latitude AS latitude, 
       covid19_locations.longitude AS longitude
FROM covid19_ts
INNER JOIN covid19_locations ON covid19_locations.province_state = covid19_ts.province_state 
WHERE $__timeFilter(covid19_ts.observation_date)
GROUP BY day,
         confirmed,
         location,
         latitude,
         longitude 
ORDER BY day;To enter this query in Grafana, click “Edit SQL” and type it in manually (or copy-paste from the above):And then in the WorldMap configuration, we set the field mapping to match our query results:And, once completed, we see the impact of the COVID-19 pandemic on the Grafana WorldMap panel:Spread of COVID-19 globally, visualized in GrafanaHow rapidly is COVID-19 spreading near me?As mentioned earlier, the Johns Hopkins dataset includes cumulative data. For example, an entry for March 2, 2020 contains all cases that have beenconfirmedat that point in time. Suppose we wanted to learn the rate at which cases have been identified near a specific location.TimescaleDB includes a feature calledcontinuous aggregates. A continuous aggregate recomputes a query automatically at user-specified time intervals and maintains the results into a table. Thus, instead of everyone running an aggregation query each time, the database can run a common aggregation periodically in the background, and users can query the results of the aggregation. Continuous aggregates should improve database performance and query speed for common calculations.In our case, we want to maintain a continuous aggregation for the daily change in confirmed cases. Let's look at this continuous aggregation query:CREATE VIEW daily_change
WITH (timescaledb.continuous, timescaledb.refresh_lag = '-2 days')
AS
SELECT
  country_region,
  province_state,
  time_bucket('2 days', observation_date) as bucket,
  first(confirmed, observation_date) as yesterday,
  last(confirmed, observation_date) as today,
  last(confirmed, observation_date) - first(confirmed, observation_date) as change
FROM
  covid19_ts
GROUP BY country_region, province_state, bucket;The first line of this query creates a continuous aggregation nameddaily_change. We then select atime_bucketwith an interval of two days calculated using thecovid19_ts.observation_datefield.In our table, we'll create ayesterdayandtodaycolumn, as well as achangecolumn that represents the delta between the two.Under normal circumstances with large amounts of data, TimescaleDB calculates continuous aggregates in the background, but if you want to see the result immediately, you can force it:ALTER VIEW daily_change SET (timescaledb.refresh_interval = '1 hour');
ALTER VIEW daily_change SET (timescaledb.refresh_lag = '-2 days');
REFRESH MATERIALIZED VIEW daily_change;We can run this continuous aggregation with a simple SQL query:SELECT * FROM daily_change;The result of that query should look something like this:day           | yesterday | today  | change |                  location                   
------------------------+-----------+--------+--------+---------------------------------------------
 2020-01-22 00:00:00+00 |         2 | [null] | [null] | 
 2020-01-22 00:00:00+00 |         1 |      9 |      8 | Anhui
 2020-01-22 00:00:00+00 |    [null] | [null] | [null] | Australia
 2020-01-22 00:00:00+00 |        14 |     22 |      8 | Beijing
 2020-01-22 00:00:00+00 |    [null] | [null] | [null] | Brazil
 2020-01-22 00:00:00+00 |         6 |      9 |      3 | Chongqing
 2020-01-22 00:00:00+00 |    [null] | [null] | [null] | Colombia
 2020-01-22 00:00:00+00 |         1 |      5 |      4 | Fujian
 2020-01-22 00:00:00+00 |    [null] |      2 | [null] | Gansu
 2020-01-22 00:00:00+00 |        26 |     32 |      6 | GuangdongWe can also incorporate the continuous aggregation in other queries, such as the one we used earlier to see the rate of change in confirmed cases near Seattle, Washington.SELECT bucket, 
       yesterday, 
       today, 
       change, 
       province_state, 
       country_region
FROM daily_change
WHERE province_state IN (
    SELECT province_state
    FROM covid19_locations
    WHERE ST_Distance(location_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-122.3321, 47.6062), 4326), 2163)) < 75000
)
GROUP BY bucket, 
         yesterday, 
         today,  
         change, 
         province_state, 
         country_region
ORDER BY bucket;Our output should look like this:bucket         | yesterday | today | change |    province_state    | country_region 
------------------------+-----------+-------+--------+----------------------+----------------
 2020-02-29 00:00:00+00 |         1 |     2 |      1 | Snohomish County, WA | US
 2020-03-02 00:00:00+00 |         4 |     6 |      2 | Snohomish County, WA | US
 2020-03-02 00:00:00+00 |        14 |    21 |      7 | King County, WA      | US
 2020-03-04 00:00:00+00 |         8 |     8 |      0 | Snohomish County, WA | US
 2020-03-04 00:00:00+00 |        31 |    31 |      0 | King County, WA      | US
(5 rows)ConclusionData gives us insight into the world around us and understanding data enables us to communicate these insights to many people. We encourage you to explore the dataset and share your dashboards and insights on Twitter using#Covid19Data(@-mention@TimescaleDBif you have questions or comments).If you need help using TimescaleDB or writing visualizations of your own, follow theHello, Timescale! tutorial. And,join our Slackto ask questions and get help from the global community of Timescale users (our engineering team is also active on all channels).Finally, no matter where you are, take prudent actions to prepare yourself and your family. Please follow the guidelines and instructionsfrom the World Health Organization.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/charting-the-spread-of-covid-19-using-timescale/
2020-01-15T21:26:35.000Z,TimescaleDB 1.6: Data Retention Policies for Continuous Aggregates,"New release allows you to save resources and storage space while maintaining continuous aggregatesWe are excited to announce the availability ofTimescaleDB 1.6today which includes significant updates to continuous aggregates among other enhancements.In particular, this release introduces the ability to use a data retention policy with continuous aggregates (a feature used to speed up queries that aggregate over time). Data retention policies are useful when you are looking to analyze large datasets without accumulating the additional storage costs.Within this post, we’ll briefly explain continuous aggregates - and the role they play in time-series data - and then dive into the specifics of what TimescaleDB 1.6 brings in terms of enhancements. But first, we want to thank TimescaleDB community members@optijon,@acarrea42,@ChristopherZellermann, and@SimonDelamarefor contributing to this release. Your feedback is always appreciated!Remind me about continuous aggregates?At a high level, by using TimescaleDBcontinuous aggregates, users can speed up queries that aggregate over time which is particularly useful for a variety of use cases such as powering real-time dashboards and deriving value from IoT sensor data.With TimescaleDB’s continuous aggregates, the view is refreshed automatically in the background as new data is added, or old data is modified. This feature is unique compared to other solutions in the market because it has the ability to properly track out-of-order data. The continuous aggregate will automatically recompute on out-of-order data without adding any maintenance burden to your database (i.e. it will not slow downINSERToperations).We introduced continuous aggregates last spring with the release of TimescaleDB 1.3 and have made severalimprovements since. If you are interested in an in-depth look at how continuous aggregates work, seeContinuous aggregates: faster queries with automatically maintained materialized views.Option to turn off data invalidationThis release includes the capability to change the limitations for data invalidation, so you can drop raw datawithoutimpacting your continuous aggregate. By turning invalidation off, you will have the option to use a data retention policy (more on this below).In case you aren’t familiar, data invalidation is a process that monitors for changes in historical data. When using data invalidation with continuous aggregates, it changes the presentation of the materialized view.In other words, if you have a lot of data marked as invalid, the background worker has to go back through and recompute the invalid data. This process takes a lot longer and consumes more resources and can slow down the performance of your continuous aggregate.With TimescaleDB 1.6, the user has the option to turn data invalidation off for data older than a certain age. When data invalidation is turned off, the view is freed from the underlying data, allowing those data points to exist as they are without worrying about changes to the data invalidating the view. As a result, refresh jobs will complete faster since they are only dealing with new data as part of the refresh, and you will have the option to implement a data retention policy.Use a data retention policy to remove underlying dataWhen data reaches a certain age, you are able to remove it manually or via adata retentionpolicy. Essentially, you have the flexibility to control the amount of data stored and allow data to “age out” after it has served its purpose.However, prior to TimescaleDB 1.6, performingdrop_chunkswasn’t optimized for use with continuous aggregates. If you performed this function, the continuous aggregate would drop all data associated with any chunks dropped from the raw hypertable. As a result, users could not purge any underlying data once a continuous aggregate was established.For example, say you are collecting metrics on a per second basis for real-time monitoring. You spread your CPU consumption across 300 compute instances and have 31,536,000 data points per year, per instance. At some point, you’d like to roll this up to 5 minute / 1 hour / 24 hour averages for analytics purposes. You’d use a continuous aggregate to do this, but need to store all the underlying data - which would create added storage expenses and hinder downsampling (i.e. the act of applying an aggregate function to roll up a very granular data set to a more coarse grained set of data to enable analytics).Now, here’s the good news! With this release, you can create a continuous aggregate and remove the underlying data using data retention policy without impacting the materialized view. And you achieve this by turning off data invalidation. This enables you to leverage downsampling by removing underlying data even after it’s rolled up.Essentially, this will allow users to save space on storage since they can choose to leverage data retention when using continuous aggregates.What if you need to retain all your old data?If you can’t leverage data retention policies because you need to retain all your raw data, remember that TimescaleDB native compression can alternatively (and also) be used to  substantially save on storage costs. (And in fact, continuous aggregations and data retention policies can be combinedwithnative compression, when you might want both!)You can learn more about compression by reading “Building columnar compression in a row-oriented database” and by visiting thedocs pages.Next stepsTo recap, TimescaleDB 1.6 will allow for more efficient storage strategy around downsampling, and more efficient use of resources. Additionally, you will achieve faster build and refresh times which leads to less worker contention.(View the Release Notes here.)Ready to try it out? Follow these upgradeinstructions. If you are brand new to TimescaleDB, get startedhere.For more information on getting started with continuous aggregates, check out ourdocs pages. We will also be publishing a blog on downsampling next week and will let you know when that’s live.If you have any questions along the way, we are always available via our communitySlackchannel.[COMING SOON: As a final note, we plan to moveautomated data retention policies and automated hypertable data reordering on diskto the Timescale Community version with the release of TimescaleDB 1.7. We will share more information in conjunction with that release.]Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescaledb-1-6-data-retention-policies-for-continuous-aggregates/
2020-05-07T15:11:33.000Z,Achieving the Best of Both Worlds:  Ensuring Up-To-Date Results With Real-Time Aggregation,"Real-time aggregates (released with TimescaleDB 1.7) build on continuous aggregates' ability to increase query speed and optimize storage. Learn what's new, details about how they work, and how to get started.One constant across all time-series use cases is data: metrics, logs, events, sensor readings; IT and application performance monitoring, SaaS applications, IoT, martech, fintech, and more.  Lots (and lots) of data. What’s more, it typically arrivescontinuously.This need to handle large volumes of constantly generated data motivated some of our earliest TimescaleDB architectural decisions, such as its use of automated time-based partitioning and local-only indexing to achieve high insert rates.  And last year, we added type-specific columnar compression to significantly shrink the overhead involved in storing all of this data (often by 90% or higher –see our technical description and benchmarking results).And another key capability in TimescaleDB, which is the focus of this post, has beencontinuous aggregates, which we firstintroducedin TimescaleDB 1.3.  Continuous aggregates allow one to specify a SQL query that continually processes raw data into a so-called materialized table.Continuous aggregates are somewhat similar to materialized views in databases, but unlike a materialized view (as inPostgreSQL), continuous aggregates do not need to be refreshed manually; the view will be refreshed automatically in the background as new data is added, or old data is modified. Additionally, TimescaleDB does not need to re-calculate all of the data on every refresh. Only new and/or invalidated data will be calculated. And since this re-aggregation is automatic – it executes as a background job at regular intervals – this process doesn’t add any maintenance burden to your database.This is where most database or streaming systems that offer continuous aggregates or continuous queries give up.  We knew we could do better.Enter Real-Time Aggregation, introduced in TimescaleDB 1.7 (see our release blog).Quick Background on Continuous AggregatesThe benefit of continuous aggregations are two fold:Query performance.By executing queries against pre-calculated results, rather than the underlying raw data,  continuous aggregates can significantly improve query performance.Storage savings withdownsampling.Continuous aggregates are often combined with data retention policies for better storage management.  Raw data can be continually aggregated into a materialized table, and dropped after it reaches a certain age.  So the database may only store some fixed period of raw data (say, one week), yet store aggregate data for much longer.Consider the following example, collecting system metrics around CPU usage and storing it in a CPU metrics hypertable, where each row includes a timestamp, hostname, and 3 metrics around CPU usage (usage_user, usage_system, usage_iowait).We collect these statistics every second per server.time              | hostname |     usage_user     |    usage_system     |    usage_iowait
-------------------------------+----------+--------------------+---------------------+---------------------
2020-05-06 02:32:34.627143+00 | host0    | 0.5378765249290502 |  0.2958572490961302 | 0.10685818344495246
2020-05-06 02:32:34.627143+00 | host1    | 0.3175958910709298 |  0.7874926624954846 | 0.16615243032654803
2020-05-06 02:32:34.627143+00 | host2    | 0.4788377981501064 | 0.18277343256546175 |  0.7183967491020162So a query that wants to compute the per-hourly histogram of usage consumption over the course of 7 days for 10 servers will process 10 servers * 60 seconds * 60 minutes * 24 hours * 7 days= 6,048,000 rows of data.On the other hand, if we pre-compute a histogram per hour, then the same query on the continuous aggregate table will only need to process 10 servers * 24 hours * 7 days = 1680 rows of data.But pre-computed results in the continuous aggregate view will lag behind the latest data, as the materialization only runs at scheduled intervals.  So, both to more cheaply handle out-of-order data and to avoid excessive load, there is typically somerefresh lagbetween the raw data and when it’s materialized.  In fact, this refresh lag is configurable in TimescaleDB, such that the continuous aggregation engine will not materialize data that’s newer than the refresh lag.(Slightly more specifically, if we compute aggregations across sometime bucket, such as hourly, then each hourly interval has a start time and end time.  TimescaleDB will only materialize data when its corresponding aggregation interval’send timeis older than the refresh lag. So, if we are doing hourly rollups with 30 minute refresh lag, then we’d only perform the materialized aggregation from, say, 2:00am - 3:00amafter2:30pm.)So, on one hand, using a continuous aggregate view has cut down the amount of data we process at query time by 3600x (i.e., from more than 6 million rows to fewer than 2000).  But, in this view, we’re often missing the last hour or so of data.While you could just make the refresh lag smaller and smaller to workaround this problem, it comes at the cost of higher and higher load; unless these aggregates are recomputed oneverynew insert (expensive!), they’re fundamentally always stale.Introducing Real-Time AggregationWith real-time aggregation, when you query a continuous aggregate view, rather than just getting the pre-computed aggregate from the materialized table, the query will transparently combine this pre-computed aggregate with raw data from the hypertable that’s yet to be materialized.  And, by combining raw and materialized data in this way, you get accurate and up-to-date results, while still enjoying the speedups that come from pre-computing a large portion of the result.Let’s return to the example above.  Recall that when we created hourly rollups, we set the refresh lag to 30 minutes, so our continuous aggregate view will lag behind by 30-90 minutes.But, when querying a view that supports real-time aggregation, the single query as before for hourly data across the past week will process and combine the results from two tables:Materialized table: 10 servers * (22 hours + 24 hours * 6 days) = 1660 rowsRaw data: 10 servers * 60 seconds * 90 minutes = 54,000 rowsSo now, with these “back of the envelope” calculations, we’ve processed a total of 55,660 rows, still well below the 6 million from before. Moreover, the last 90 minutes of data are more likely to already be memory resident for even better performance, given the database page caching already happening for recent data.Real-time aggregates allow you to query your pre-calculated dataandnewer, not yet materialized ""raw"" dataThe above illustration shows this in practice. The database internally maintains acompletion thresholdas metadata, which records the point-in-time to which all previous records from the raw table have been materialized.  This completion threshold lags behind therefresh lagwe discussed earlier, and gets updated by the database engine whenever a background task updates the materialized view.(In fact, it’s a bit more complicated given TimescaleDB’s ability to handle late data that gets written after some time region has already been materialized, i.e., behind the completion threshold.  But we’re going to ignore how TimescaleDB tracks invalidation regions in this post.)So now when processing our query covering the interval , the database engine will conceptually take a UNION ALL between results from the materialized table starting atnow() - interval '7 days'up to the completion threshold, with results from the raw table from the completion threshold up tonow().But rather than just describe this behavior, let’s walk through a concrete example and compare our query times without continuous aggregates, with vanilla continuous aggregates, and with real-time aggregation enabled.These capabilities were developed by Timescale engineers:Sven Klemm,Matvey Arye,Gayathri Ayyapan,David Kohn, andJosh Lockerman.Testing Real-Time AggregationIn the following, I’ve created a TimescaleDB 1.7 instance viaManaged Service for TimescaleDB(specially, an “basic-100-compute-optimized” instance with PostgreSQL 12, 4 vCPU, and 100GB SSD storage), and then created the following hypertable:$ psql postgres://[email protected]:26479/defaultdb?sslmode=require

=> CREATE TABLE cpu (
      time TIMESTAMPTZ,
      hostname TEXT,
      usage_user FLOAT,
      usage_system FLOAT,
      usage_iowait FLOAT
   );

=> SELECT create_hypertable ('cpu', 'time', 
      chunk_time_interval => interval '1d');I’m now going to load the hypertable with 14 days of synthetic data (which is created with the following INSERT statement):=> INSERT INTO cpu (
   SELECT time, hostname, random(), random(), random()
      FROM generate_series(NOW() - interval '14d', NOW(), '1s') AS time
      CROSS JOIN LATERAL (
         SELECT 'host' || host_id::text AS hostname 
            FROM generate_series(0,9) AS host_id
      ) h
   );Okay, so that inserted 12,096,010 rows of synthetic data into our hypertable of the following format, stretching from 2:32am UTC on April 22 to 2:32am UTC on May 6:=> SELECT * FROM cpu ORDER BY time DESC LIMIT 3;

             time              | hostname |     usage_user     |    usage_system     |    usage_iowait     
-------------------------------+----------+--------------------+---------------------+---------------------
 2020-05-06 02:32:34.627143+00 | host0    | 0.5378765249290502 |  0.2958572490961302 | 0.10685818344495246
 2020-05-06 02:32:34.627143+00 | host1    | 0.3175958910709298 |  0.7874926624954846 | 0.16615243032654803
 2020-05-06 02:32:34.627143+00 | host2    | 0.4788377981501064 | 0.18277343256546175 |  0.7183967491020162


=> SELECT min(time) AS start, max(time) AS end FROM cpu;

-[ RECORD 1 ]------------------------
start | 2020-04-22 02:32:34.627143+00
end   | 2020-05-06 02:32:34.627143+00Let’s now create a continuous aggregate view on this table with hourlyhistograms:=> CREATE VIEW cpu_1h 
   WITH (timescaledb.continuous, 
         timescaledb.refresh_lag = '30m',
         timescaledb.refresh_interval = '30m')
   AS
      SELECT 
         time_bucket('1 hour', time) AS hour,
         hostname, 
         histogram(usage_user, 0.0, 1.0, 5) AS hist_usage_user,
         histogram(usage_system, 0.0, 1.0, 5) AS hist_usage_system,
         histogram(usage_iowait, 0.0, 1.0, 5) AS hist_usage_iowait
      FROM cpu
      GROUP BY hour, hostname;By default, queries to this view use these real-time aggregation features.  If you want to disable real-time aggregation, setmaterialized_only = truewhen creating the view or by later ALTERing the view.  (SeeAPI docs here.)Now, the job scheduling framework will start to asynchronously process this view, which we can see in ourinformational view.  (You can alsomanually forcethe materialization to occur if needed.)=> SELECT * FROM timescaledb_information.continuous_aggregate_stats;

- [ RECORD 1 ]
view_name              | cpu_1h
completed_threshold    | 2020-05-06 02:00:00+00
invalidation_threshold | 2020-05-06 02:00:00+00
job_id                 | 1000
last_run_started_at    | 2020-05-06 02:34:08.300524+00
last_successful_finish | 2020-05-06 02:34:09.04923+00
last_run_status        | Success
job_status             | Scheduled
last_run_duration      | 00:00:00.748706
next_scheduled_run     | 2020-05-06 03:04:09.04923+00
total_runs             | 17
total_successes        | 17
total_failures         | 0
total_crashes          | 0From this data, we see that the materialized view includes data up to 2:00am on May 6, while from above we’ve learned that the raw data goes up to 2:32am.Let’s try our query directly on the raw table, and use an EXPLAIN ANALYZE to both show the database plan, as well as actually execute the query and collect timing information.  (Note that in many use cases, one would offset queries fromnow() - <some interval>. But to ensure that we use identical datasets in our subsequent analysis, we explicitly select the interval offset from the dataset’s last timestamp.)=> EXPLAIN (ANALYZE, COSTS OFF)
   SELECT 
      time_bucket('1 hour', time) AS hour,
      hostname, 
      histogram(usage_user, 0.0, 1.0, 5) AS hist_usage_user,
      histogram(usage_system, 0.0, 1.0, 5) AS hist_usage_system,
      histogram(usage_iowait, 0.0, 1.0, 5) AS hist_usage_iowait
   FROM cpu
   WHERE time > '2020-05-06 02:32:34.627143+00'::timestamptz - interval '7 days'
   GROUP BY hour, hostname
   ORDER BY hour DESC;

QUERY PLAN             
----------------------------------------------------------------
 Finalize GroupAggregate (actual time=1859.306..1862.331 rows=1690 loops=1)
   Group Key: (time_bucket('01:00:00'::interval, cpu.""time"")), cpu.hostname
   ->  Gather Merge (actual time=1841.735..1849.604 rows=1881 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Sort (actual time=1194.162..1194.222 rows=627 loops=3)
               Sort Key: (time_bucket('01:00:00'::interval, cpu.""time"")) DESC, cpu.hostname
               Sort Method: quicksort  Memory: 25kB
               Worker 0:  Sort Method: quicksort  Memory: 274kB
               Worker 1:  Sort Method: quicksort  Memory: 274kB
               ->  Partial HashAggregate (actual time=1193.198..1193.594 rows=627 loops=3)
                     Group Key: time_bucket('01:00:00'::interval, cpu.""time""), cpu.hostname
                     ->  Parallel Custom Scan (ChunkAppend) on cpu (actual time=9.840..716.952 rows=2016000 loops=3)
                           Chunks excluded during startup: 7
                           ->  Parallel Seq Scan on _hyper_1_14_chunk (actual time=14.751..199.098 rows=864000 loops=1)
                                 Filter: (""time"" > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))
                           ->  Parallel Seq Scan on _hyper_1_13_chunk (actual time=14.749..201.100 rows=864000 loops=1)
                                 Filter: (""time"" > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))
                           ->  Parallel Seq Scan on _hyper_1_12_chunk (actual time=0.025..182.591 rows=864000 loops=1)
                                 Filter: (""time"" > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))
                           ->  Parallel Seq Scan on _hyper_1_11_chunk (actual time=0.031..182.812 rows=864000 loops=1)
                                 Filter: (""time"" > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))
                           ->  Parallel Seq Scan on _hyper_1_10_chunk (actual time=0.035..183.918 rows=864000 loops=1)
                                 Filter: (""time"" > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))
                           ->  Parallel Seq Scan on _hyper_1_9_chunk (actual time=0.019..184.416 rows=864000 loops=1)
                                 Filter: (""time"" > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))
                           ->  Parallel Seq Scan on _hyper_1_8_chunk (actual time=0.823..91.605 rows=386225 loops=2)
                                 Filter: (""time"" > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))
                                 Rows Removed by Filter: 45775
                           ->  Parallel Seq Scan on _hyper_1_15_chunk (actual time=0.022..20.277 rows=91550 loops=1)
                                 Filter: (""time"" > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))

 Planning Time: 1.917 ms
 Execution Time: 1921.753 msNote that TimescaleDB’s constraint exclusion excluded 7 of the chunks from being queried given the WHERE predicate (as the query was for the last 7 days of the 14 day dataset), then processed the query on the remaining 8 chunks (performing a scan over 6,048,000 rows) using two parallel workers.  The query in total took just over 1.9 seconds.Now let’s try the query on our materialized table, first turning off real-time aggregation just for this experiment:=> ALTER VIEW cpu_1h set (timescaledb.materialized_only = true);First, let’s look at the table definition, which defines a SELECT on the materialized view with the specified GROUP BYs.  But we also see that each of the histograms calls “finalize_agg.”  TimescaleDB doesn’t precisely pre-compute and store the exact answer that’s specified in the query, but rather apartial aggregatethat is then “finalized” at query time, which will allow for greater parallelization and rebucketing at query time (in a future release).\d+ cpu_1h;

                                          View ""public.cpu_1h""
      Column       |           Type           | Collation | Nullable | Default | Storage  | Description 
-------------------+--------------------------+-----------+----------+---------+----------+-------------
 hour              | timestamp with time zone |           |          |         | plain    | 
 hostname          | text                     |           |          |         | extended | 
 hist_usage_user   | integer[]                |           |          |         | extended | 
 hist_usage_system | integer[]                |           |          |         | extended | 
 hist_usage_iowait | integer[]                |           |          |         | extended | 

View definition:
 SELECT _materialized_hypertable_2.hour,
    _materialized_hypertable_2.hostname,
    _timescaledb_internal.finalize_agg('histogram(double precision,double precision,double precision,integer)'::text, NULL::name, NULL::name, '{{pg_catalog,float8},{pg_catalog,float8},{pg_catalog,float8},{pg_catalog,int4}}'::name[], _materialized_hypertable_2.agg_3_3, NULL::integer[]) AS hist_usage_user,
    _timescaledb_internal.finalize_agg(...) AS hist_usage_system,
    _timescaledb_internal.finalize_agg(...) AS hist_usage_iowait
   FROM _timescaledb_internal._materialized_hypertable_2
  GROUP BY _materialized_hypertable_2.hour, _materialized_hypertable_2.hostname;Now let’s run the query with vanilla continuous aggregates enabled:=> EXPLAIN (ANALYZE, COSTS OFF)
   SELECT * FROM cpu_1h
   WHERE hour > '2020-05-06 02:32:34.627143+00'::timestamptz - interval '7 days'
   ORDER BY hour DESC;

QUERY PLAN
----------------------------------------------------------------
 Sort (actual time=3.218..3.312 rows=1670 loops=1)
   Sort Key: _materialized_hypertable_2.hour DESC
   Sort Method: quicksort  Memory: 492kB
   ->  HashAggregate (actual time=1.943..2.891 rows=1670 loops=1)
         Group Key: _materialized_hypertable_2.hour, _materialized_hypertable_2.hostname
         ->  Custom Scan (ChunkAppend) on _materialized_hypertable_2 (actual time=0.064..0.688 rows=1670 loops=1)
               Chunks excluded during startup: 1
               ->  Seq Scan on _hyper_2_17_chunk (actual time=0.063..0.590 rows=1670 loops=1)
                     Filter: (hour > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))
                     Rows Removed by Filter: 270

 Planning Time: 0.645 ms
 Execution Time: 3.461 msJust 4 milliseconds, after a scan of 1,670 rows in the materialized hypertable.  And let’s look at the most recent 3 rows returned for a specific host:=> SELECT hour, hostname, hist_usage_user
    FROM cpu_1h
    WHERE hour > '2020-05-06 02:32:34.627143+00'::timestamptz - interval '7 days'         
       AND hostname = 'host0'
    ORDER BY hour DESC LIMIT 3;

          hour          | hostname |      hist_usage_user      
------------------------+----------+---------------------------
 2020-05-06 01:00:00+00 | host0    | {0,781,676,712,719,712,0}
 2020-05-06 00:00:00+00 | host0    | {0,736,714,776,689,685,0}
 2020-05-05 23:00:00+00 | host0    | {0,714,759,715,692,720,0}Note that the last record is from the 1:00am - 2:00am hour.Now let’s re-enable real-time aggregation and try the same query, first showing how the real-time aggregation is defined as a UNION ALL between the materialized and raw data.=> ALTER VIEW cpu_1h set (timescaledb.materialized_only = false);

=> \d+ cpu_1h;

                                          View ""public.cpu_1h""
      Column       |           Type           | Collation | Nullable | Default | Storage  | Description 
-------------------+--------------------------+-----------+----------+---------+----------+-------------
 hour              | timestamp with time zone |           |          |         | plain    | 
 hostname          | text                     |           |          |         | extended | 
 hist_usage_user   | integer[]                |           |          |         | extended | 
 hist_usage_system | integer[]                |           |          |         | extended | 
 hist_usage_iowait | integer[]                |           |          |         | extended | 

View definition:
 SELECT _materialized_hypertable_2.hour,
    _materialized_hypertable_2.hostname,
    _timescaledb_internal.finalize_agg(...) AS hist_usage_user,
    _timescaledb_internal.finalize_agg(...) AS hist_usage_system,
    _timescaledb_internal.finalize_agg(...) AS hist_usage_iowait
   FROM _timescaledb_internal._materialized_hypertable_2
  WHERE _materialized_hypertable_2.hour < COALESCE(_timescaledb_internal.to_timestamp(_timescaledb_internal.cagg_watermark(1)), '-infinity'::timestamp with time zone)
  GROUP BY _materialized_hypertable_2.hour, _materialized_hypertable_2.hostname
UNION ALL
 SELECT time_bucket('01:00:00'::interval, cpu.""time"") AS hour,
    cpu.hostname,
    histogram(cpu.usage_user, 0.0::double precision, 1.0::double precision, 5) AS hist_usage_user,
    histogram(cpu.usage_system, 0.0::double precision, 1.0::double precision, 5) AS hist_usage_system,
    histogram(cpu.usage_iowait, 0.0::double precision, 1.0::double precision, 5) AS hist_usage_iowait
   FROM cpu
  WHERE cpu.""time"" >= COALESCE(_timescaledb_internal.to_timestamp(_timescaledb_internal.cagg_watermark(1)), '-infinity'::timestamp with time zone)
  GROUP BY (time_bucket('01:00:00'::interval, cpu.""time"")), cpu.hostname;


=> EXPLAIN (ANALYZE, COSTS OFF)
   SELECT * FROM cpu_1h
   WHERE hour > '2020-05-06 02:32:34.627143+00'::timestamptz - interval '7 days'
   ORDER BY hour DESC;

QUERY PLAN               
----------------------------------------------------------------
 Sort (actual time=20.871..21.055 rows=1680 loops=1)
   Sort Key: _materialized_hypertable_2.hour DESC
   Sort Method: quicksort  Memory: 495kB
   ->  Append (actual time=1.842..20.536 rows=1680 loops=1)
         ->  HashAggregate (actual time=1.841..2.789 rows=1670 loops=1)
               Group Key: _materialized_hypertable_2.hour, _materialized_hypertable_2.hostname
               ->  Custom Scan (ChunkAppend) on _materialized_hypertable_2 (actual time=0.105..0.580 rows=1670 loops=1)
                     Chunks excluded during startup: 1
                     ->  Index Scan using _hyper_2_17_chunk__materialized_hypertable_2_hour_idx on _hyper_2_17_chunk (actual time=0.104..0.475 rows=1670 loops=1)
                           Index Cond: ((hour < COALESCE(_timescaledb_internal.to_timestamp(_timescaledb_internal.cagg_watermark(1)), '-infinity'::timestamp with time zone)) AND (hour > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval)))
         ->  HashAggregate (actual time=17.641..17.655 rows=10 loops=1)
               Group Key: time_bucket('01:00:00'::interval, cpu.""time""), cpu.hostname
               ->  Custom Scan (ChunkAppend) on cpu (actual time=0.165..12.297 rows=19550 loops=1)
                     Chunks excluded during startup: 14
                     ->  Index Scan using _hyper_1_15_chunk_cpu_time_idx on _hyper_1_15_chunk (actual time=0.163..9.723 rows=19550 loops=1)
                           Index Cond: (""time"" >= COALESCE(_timescaledb_internal.to_timestamp(_timescaledb_internal.cagg_watermark(1)), '-infinity'::timestamp with time zone))
                           Filter: (time_bucket('01:00:00'::interval, ""time"") > ('2020-05-06 02:32:34.627143+00'::timestamp with time zone - '7 days'::interval))

 Planning Time: 3.532 ms
 Execution Time: 22.905 msStill very fast at just over 26 milliseconds (scanning 1,670 materialized rows and 19,550 raw rows), and now the results:=> SELECT hour, hostname, hist_usage_user
   FROM cpu_1h
WHERE hour > '2020-05-06 02:32:34.627143+00'::timestamptz - interval '7 days'
      AND hostname = 'host0'
   ORDER BY hour DESC LIMIT 3;

          hour          | hostname |      hist_usage_user      
------------------------+----------+---------------------------
 2020-05-06 02:00:00+00 | host0    | {0,384,388,385,400,398,0}
 2020-05-06 01:00:00+00 | host0    | {0,781,676,712,719,712,0}
 2020-05-06 00:00:00+00 | host0    | {0,736,714,776,689,685,0}Unlike when we were processing the materialized table without the real-time aggregation, we have up-to-date data with data from the 2:00 - 3:00am hour.  This is because the materialized table didn’t have data from the last hour, while the real-time aggregation was able to compute that result from the raw data at query time.  You can also notice that there is less data in the final row (namely, each histogram bucket has about half the counts as the prior rows), as this final row was the aggregation of 32 minutes of raw data, not a full hour.You can also observe these two stages of real-time aggregation in the above query plan:  the materialized hypertable is processed in the first section viaCustom Scan (ChunkAppend) on _materialized_hypertable_2, while the underlying raw hypertable is processed in the second section viaCustom Scan (ChunkAppend) on cpu, and each processes only before or after the offset specified by the completion threshold (shown with_timescaledb_internal.cagg_watermark(1)in the plan).So, in summary:  a complete, up-to-date aggregate over the data, both at a fraction of the latency of querying the raw data, and avoiding the excessive overhead of schemes that update materalizations through per-row or per-statement triggers.Query TypeLatencyFreshnessRaw Data1924 msUp-to-dateContinuous Aggregates4 msLags up to 90 minutesReal-Time Aggregation26 msUp-to-dateContinuous aggregates and real-time aggregation for the win!ConclusionsWhat motivated us to build TimescaleDB is the firm belief that time-series use cases need a best-in-class, flexible time-series database, with advanced capabilities specifically designed for time-series workloads.  We developed real-time aggregation for time-series use cases such as devops monitoring, real-time analytics, and IoT, where fast queries over high-volume workloads and accurate, real-time results really matter.Real-time aggregation joins a number of advanced capabilities in TimescaleDB around data lifecycle management and time-series analytics, including automated data retention, data reordering, native compression, downsampling, and traditional continuous aggregates.And,there’s still much more to come. Keep an eye out for our much-anticipated TimescaleDB 2.0 release, which introduces horizontal scaling to TimescaleDB for terabyte to petabyte workloads.Want to check out real-time aggregation?Ready to dig in? Check out ourdocs.Brand new to TimescaleDB?  Get startedhere.If you have any questions along the way, we’re always available via ourcommunity Slack(we’re@mikeand@sven, come say hi 👋).And, if you are interested in keeping up-to-date with future TimescaleDB releases,sign up for our Release Notes.  It’s low-traffic, we promise.Until next time, keep it real!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/achieving-the-best-of-both-worlds-ensuring-up-to-date-results-with-real-time-aggregation/
2019-09-25T16:52:32.000Z,How to Quickly Build SQL Dashboards for Time-Series Data,"Achieve automated materialized views using TimescaleDB continuous aggregates.Time-series dataprovides significant value to organizations because it enables them to analyze important real-time and historical metrics. However, data is valuable only if it’s easy to access. That’s where being able to build SQL dashboards that run repetitive analytical queries becomes a force multiplier for organizations looking to expose their time-series data across teams.In this post, we will discuss why you should use TimescaleDBcontinuous aggregatesto speed up queries that aggregate over time. Additionally, we will walk through asample applicationthat leverages this feature specific to TimescaleDB.Why continuous aggregates?Powering real-time dashboards is one of the top use cases for TimescaleDB. Dashboards are unique in that they are often pre-built and run repetitive queries. However, any database powering dashboards also needs to be able to support ad-hoc queries since this increases the agility of creating, updating, and adjusting dashboards.PostgreSQL excels at these ad-hoc queries by offering the flexible SQL interface. However, as data sizes increase, PostgreSQL tends to fall over. That’s where TimescaleDB comes in!With automated partitioning, TimescaleDB can optimize analytical time-series queries so that less data is read from disk. We’ve also built continuous aggregates, which further reduce disk throughput and compute requirements when running historical aggregate queries. By combining the inherent flexibility of SQL with the advanced capabilities of TimescaleDB, you can perform the ad-hoc queries needed to define your dashboards, as well as efficiently run repetitive analytical queries to power ongoing live dashboards.Should I be using continuous aggregates?We built continuous aggregates to significantly speed up repetitive queries that calculate aggregates over periods of time. It is not really meant for point queries, e.g. queries that search for values at a single or short period of time. Essentially, a continuous aggregates populates a materialized view that stores aggregates on a scheduled basis. By storing aggregates, it minimizes the computation required to run an aggregate query and the amount of data that needs to be read off disk.TimescaleDB continuous aggregates do have some specific characteristics that you should be aware of as you decide if you want to use this feature.Continuous aggregates are scheduled.You define how up-to-date you want your aggregates to be. If you want the aggregates to be updated at the same time a value is written into the original hypertable, you’ll end up incurring write amplification at the time of insert. This will result in lower inserts per second. We typically recommend users looking for fast insert performance to allow aggregates to lag behind the bleeding insert edge.Continuous aggregates support out-of-order inserts.Some systems that implement continuous aggregates actually ignore out of order inserts. In TimescaleDB, when the scheduled job runs to update the continuous aggregate, we also check for any out of order inserts and update those associated aggregates.The continuous aggregate itself is actually a hypertable.You don’t have to worry about how to scale that particular table.Walking through a sample application leveraging continuous aggregatesLet’s walk through how continuous aggregates work be exploring a sample application. We first introduced this application in ourtime_bucket() analytics post, where we showed how you could build flexible graphs using TimescaleDB’stime_bucket()function.The sample application is written in Python. It essentially scrapes the Open AQ (air quality) API, parses the results, and stores all measurements collected from air quality sensors for all cities in Great Britain. You can check out the codehere.The Grafana dashboard locatedheredescribes step-by-step how to set up multiple continuous aggregates. To give you a quick sense of why continuous aggregates are really useful, here’s a query that we use in the Air Quality example:SELECT
  time_bucket('1 day', time) as bucket,
  parameter_id,
  avg(value) as avg,
  max(value) as max,
  min(value) as min
FROM
  measurements
GROUP BY bucket, parameter_id;You’ll notice that we are querying across all time and bucketing things by 1 day intervals. The computations of bucketing things by 1 day, as well as reading all that data off disk is high. The table at the point at which I’m running this query has ~700k rows. You can imagine how this query could get really slow as you add more data.Below is an example query plan. Notice how much work is required to compute this query!To speed this up, I wrote a continuous aggregate.Now, let’s try to query the same thing, but query the measurements_daily table directly. The query plan is greatly simplified in that it is no longer scanning as much data.Advanced configurations and togglesWhen we designed continuous aggregates, we wanted users to be able to toggle and configure the characteristics that they wanted to achieve from continuous aggregates. You most likely won’t need to touch a lot of the toggles, but here are a couple that you should definitely be aware of.timescaledb.refresh_interval:This configuration specifies how often you want a continuous aggregate to run. Note that when a continuous aggregate is run, it only runs on new data that it hasn’t observed before. This includes any data that came in out of order that invalidates older continuous aggregates. The value you choose here impacts how hard your background workers are working in order to calculate continuous aggregates.timescaledb.refresh_lag:This configuration specifies how far behind a continuous aggregate runs compared to the insert bleeding edge. This impacts write amplification. If you set the refresh_lag to the negative bucket width, the continuous aggregate will immediately calculate and update the underlying materialized view whenever new data is received. However, this also means that you are executing multiple writes on insert, which impacts insert performance.timescaledb.max_interval_per_job:Some of our users using continuous aggregates for the first time notice that their aggregates never populate. This is a situation where you want to take a look at this value - it controls how long a query can run in the background before we kill it. If you take a huge table and try to create a continuous aggregate on it for the first time, it’s likely that you’ll hit this barrier.The best way to check if your configurations make sense is to usetimescaledb_information.continuous_aggregate_stats. These stats will show you important statistics about your continuous aggregates.Next stepsAs you can see, continuous aggregates can be an extremely useful function when you are looking to further reduce disk throughput and compute requirements when running historical aggregate queries.If you are ready to try it out for yourself, check out thistutorial. If you are brand new to TimescaleDB, follow theinstallation instructionsor get started withManaged Service for TimescaleDB.Need help along the way? You can always reach out to us on our communitySlackchannel.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-quickly-build-dashboards-with-time-series-data/
2020-04-07T18:15:47.000Z,CTO Corner: This Month in Data (April 2020 Edition),"Here are some of the most interesting articles about databases and distributed systems we’ve found in the past month.TheTimescaleteam includes a bunch of people who care deeply about databases, distributed systems, and generally everything around data.Many of us have an academic research background – some of our first engineering hires came out of the computer science department at Princeton, where I’m also aprofessor– while others are industry veterans with deep expertise.As such, we stay connected with what’s happening in both the academic and developer communities.  So each month, I’ll list and summarize a non-exhaustive assortment of the more interesting papers and articles we’ve read over the last few weeks (and invite you to share your own).Academic PapersMillions of Tiny DatabasesThree Amazon Web Services engineers wrote this really interesting paper at USENIX’sNSDI '20conference that describes the Physalia databases, which serves as a configuration master for AWS’sElastic Block Store(EBS).  Because EBS is deployed throughout AWS and requires extremely high availability, the goal wasn’t a single monolithic database (even if a geo-replicated one, see theCAP Theorem), but instead millions of little databases to reduce the “blast radius” of hardware failures.FirecrackerAnother AWS paper from NSDI '20, Firecracker describes the next-generation virtualization/isolation architecture for AWS Lambda.  The paper talks about the various limitations that Lambda encountered in its v1 (namely, using Linux containers between functions, and VMs between accounts), and how Firecracker addresses these.  And with a memory footprint of only 3MB per microVM (vs. 130MB for QEMU VM).  And written in Rust, for you language folks!Adrian Colyer’sthe morning paperSpeaking of the above two papers, Adrian has a wonderful site and mailing list for great, highly readable summaries of recent (and sometimes not-so-recent) papers, with a particular focus on systems and occasionally AI/ML. Adrian reviewed both theMillions of Tiny DatabasesandFirecrackerpapers, and his site features dozens more.Articles from the IndustryHistory of parallelism in PostgreSQLThis post from Amit Kapila tracks parallelism in PostgreSQL (the foundation of TimescaleDB), starting in PG 9.6.  It also dives into what the future may hold for our favorite database, including for PG13 and beyond. This is something that we’ve thought about a lot ourselves, especially as we leverage parallelism and async operations in parallel queries across data nodes inmulti-node TimescaleDB.Dropbox’s new sync engineI enjoyed this post from Dropbox on the architectural implementation of their new sync engine, particularly around some of the general engineering deliberations behind their decision to completely rewrite code at the core of all of Dropbox’s desktop clients. And another bet on Rust (migrating from Python).Oldies but GoodiesHow complex systems failThis paper has long been read by aspiring computer scientists and engineers; in fact, my cofounder Ajay and I both read it in the late 1990s as undergrads taking MIT’s 6.033 course withJerry Salzer. The author, Richard Cook, wrote this from his career thinking about patient safety, which seems particularly relevant in the midst of the COVID-19 global pandemic.For a longer take,Richard Cook spokeon the subject at O’Reilly Velocity 2012.Data Domain’s Deduplication File SystemAcademic conferences often give out “Test of Time” awards 10-15 years after papers are first published. This year’sFAST '20 conferenceawarded its Test of Time to this paper from FAST 2008, which describes various data structure, layout, and caching optimizations to reduce disk access by 99% for deduplication workloads. (Data Domain was originally founded by my Princeton colleague,Kai Li, and went on to become a highly successful business unit of EMC.)Closing thoughtsI’d love to hear about what you thought of these papers, as well as get your suggestions for articles, sites, and videos we should read/watch and highlight. You can find me on Twitter (@michaelfreedman) or in theTimescale Community Slack.Until next month!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/cto-corner-april-edition/
2020-12-17T16:00:00.000Z,How to Store Time-Series Data in MongoDB and Why That’s a Bad Idea,"An insert performance up to 260 % higher, 54x faster queries, and simpler implementation when using TimescaleDB vs. MongoDB for time-series data.Time-series datahas exploded in popularity, and the value of tracking and analyzing how things change over time has become evident in every industry: DevOps and IT monitoring, industrial manufacturing, financial trading and risk management, sensor data, ad tech, application eventing, smart home systems, autonomous vehicles, and more.But 2020 has provided us with the most personal example of how time-series data collection and analysis affects our daily lives, with billions of people across the globe becoming relentless consumers of time-series data, demanding accurate and timely information about thedaily trend of various COVID-19 statistics.The biggest challenge with storing time-series data? Scale, both in collecting data and storing it. Data accumulates quickly and requires a database that can keep up with a relentless stream of data from the systems you care about. We’ve shown previously thatSQL and relational databases can reach petabyte-scale and beyond, but many developers' first inclination still goes to using a NoSQL database for their time-series data when scale is a requirement (perhaps due to thebreakthroughs made by NoSQL databases in the early 2000s?)Enter MongoDB as a time-series solution.MongoDB is among the best-known NoSQL databases, emerging at the end of the last decade to become the face of NoSQL and the foundation of a nearly$21 billion company(as of writing). MongoDB became popular as a simple document store for quickly prototyping and easily scaling web apps.But, over the years, developers have started using MongoDB to fill all sorts of database needs across various domains, including using MongoDB to store and analyze time-series data at scale.Here are a few examples of posts we’ve found on the topic of storing time-series data in MongoDB, with sources ranging from the official MongoDB blog to popular technical how-to sites, like Dev.to and Quora:Part 2of a series on storing time-series data in MongoDB on the official MongoDB blog.Post on dev.toabout how to implement time-series in MongoDB.Post onQuora, the popular Q&A platform, about the best way to store time-series data in MongoDB.But is MongoDB really the right solution for time-series data? We decided to evaluate it for ourselves, with the obvious caveat that we are the creators of a competing product. Yet, you’ll see we try to keep our analysis as fair as possible, trying multiple approaches to storing time-series data in MongoDB.Our conclusion is that while MongoDB’s JSON-like document store may make it a jack-of-all-trades type of database and perhaps a master of some (e.g., web applications), time seriesis not one of them.You’re better off going with a purpose-built time-series database for both performance and ease of use.Our Evaluation: TimescaleDB vs MongoDBFor this analysis, we evaluated MongoDB vs.TimescaleDB, the leading open-source relational database for time-series data (and our own product). Engineered atop PostgreSQL, TimescaleDB is designed for fast ingest, complex queries, and ease of use, with powerful tools for analysis, retention, and management of time-series data. Given its PostgreSQL foundation, it inherits the rock-solid reliability, tooling, and vast ecosystem of Postgres, as well as SQL, the commonly known, well-documented query language that’s popular for data analysis.We evaluated two methods of using MongoDB as a time-series database:“Mongo-naive”: a naive, document-per-event method.“Mongo-recommended”: a method recommended byMongoDB usersandMongoDB itselfthat aggregates events into hourly documents.We compared MongoDB and TimescaleDB across several dimensions:Insert (write) performanceQuery (read) performanceDisk storage footprintImplementation cost and code maintenanceHere’s a summary of our results. We go into much more detail about our methodology later in the post:The first method,which we’ll call “Mongo-naive” throughout this post, has fast writes and is extremely simple to implement but offers dismal query performance, even on simple analytical queries.The second method, which we’ll call “Mongo-recommended” throughout this post, demonstrates 550-6800% of the query performance of Mongo-naive (method 1). This performance difference makes it clear to see why it is the recommended method for storing time-series data in MongoDB, despite having slower ingest performance (1.5x slower) and more disk usage (16% more) than Mongo-naive.However, this method has worse write performance, higher storage footprint, and higher implementation complexity than both Mongo-naive and TimescaleDB. Mongo-recommended also fails to deliver good query performance on more complex time-series queries compared to TimescaleDB.More specifically, compared to MongoDB, TimescaleDB exhibits:169 % (vs. method 1) to 260 % (vs. method 2) regarding write performance10x smaller disk storage footprintthan both MongoDB methods,using 9 % of the storage footprint of Mongo-naive and 8 % of the storage footprint of Mongo-recommended.TimescaleDB has 10x smaller disk storage footprint than both MongoDB methods.200% to 5400% faster queries,performing considerably faster on complex queries, which are commonly used to analyze and monitor devices for DevOps and IoT use casesTimescaleDB showed 200% to 5400% faster queries than MongoDB during our benchmark evaluation.Simpler implementation and code maintenance, especially when compared to method 2, the method MongoDB itself recommends.Want to learn more about TimescaleDB?Create a free accountto get started with a fully managed TimescaleDB instance (100 % free for 30 days).Want to host TimescaleDB yourself?Visit our GitHubto learn more about options, get installation instructions, and more (and, as always, ⭐️  are  appreciated!)(P.S. If you’re looking for more comparisons of database performance for time-series data, read our studies comparingAWS Timestream vs. TimescaleDBandInfluxDB vs. TimescaleDB.)In the remainder of this post, we’ll outline the methodology and results used to conduct a detailed set of benchmarks that compare TimescaleDB vs. MongoDB across inserts, queries, and ease of use.MongoDB Time-Series MethodsAs mentioned, we tested two methods for storing time-series data in MongoDB, and before diving into write and read performance numbers, let’s take a moment to examine each method in a bit more detail.We’ll use the common time-series scenario of DevOps metrics (in this case, storing CPU metrics) to demonstrate how to store time-series data in MongoDB.Method 1: Document per event (a.k.a. “Mongo-naive”)The first method, which we’ll refer to as “Mongo-naive,” is straightforward: each time-series reading is stored as a document. This is the first approach that would come to mind for most developers when trying to store time-series data in MongoDB, hence the name ""naive"".So, for our given use case (see setup below) of monitoring CPU metrics, the JSON document looks like this:{
    “measurement” : “cpu”,
    “timestamp_ns” : NumberLong(“1451606420000000000”),
    “fields” : {
        “usage_user” : 98.15664166391042,
        “usage_guest_nice” : 85.53066998716453,
        …
    },
    “tags” : {
        “hostname” : “host_2019”,
        “datacenter” : “us-east-1b”,
        …
    }
}Conceptually and in implementation, this method is very simple, so it seems like a tempting route to go: batch all measurements that occur at the same time into one document along with their associated tags and store them as one document.Indeed, this approach yields very good write performance, and it is fairly easy to implement (in our tests, we sawnoticeably better results than InfluxDB reports in their comparison). However, we’ll see later that, even with indexes, the query performance with this method leaves much to be desired.Method 2: Aggregate readings hourly per device (a.k.a. “Mongo-recommended”)This second method is one thatMongoDB usersand thecompany itself recommendwhen it comes to time series, which we call “Mongo-recommended.”  The engineering idea behind this method is clever: create a document for each device, for each hour, which contains a 60 by 60 matrix representing every second of every minute in that hour. This document is then updated each time a new reading comes in, rather than doing a new document insert:{
  ""measurement"" : ""cpu"",
  ""doc_id"" : ""host_15_20160101_00"",  // for quick update lookup
  ""key_id"" : ""20160101_00"",  // YYYYMMDD_hh
  ""tags"" : {
    ""hostname"" : ""host_6"",
    ""datacenter"" : ""ap-southeast-1b"",
    ...
  },
  ""events"" : [
    [
      {
        ""timestamp_ns"" : NumberLong(""1451606420000000000""),
        ""usage_user"" : 98.15664166391042,
        ""usage_guest_nice"" : 85.53066998716453,
        ...
      },
      ... // (59 elements elided)
    ],
    ... // (59 elements elided)
  ]
}This method makes it possible to do some efficient filtering when it comes to queries but comes with a more cumbersome implementation and decreased (albeit not terrible) write performance. For example, to efficiently manage writes, the database must keep a client-side cache of which documents are already made so that a more costly “upsert” (i.e., insert if it doesn’t exist; otherwise, update) pattern is not needed.Further, while queries for this method are typically more performant, we found that designing the query in the first place requires more effort than method 1, especially when reasoning about which aggregate documents can be filtered/pruned.Finally, this approach limits the granularity of your data. In particular, if you wanted to support millisecond precision, you would have to change the design to aggregate on a minute basis, as the max document size in MongoDB (16MB) does not lend itself to further nesting.Beyond millisecond precision is probably infeasible, as MongoDB’s document size limitation means that you would probably need to create a document per second for each device, and the nesting required would make the query construction process extremely complex.Because the data in our evaluation was at the granularity of seconds, not milliseconds, and given the query performance we saw (as detailed in the next section),we ultimately decided that this method is probably the best method for comparison against TimescaleDB. (We include Mongo-naive write and read performance numbers  to show how we reached this conclusion.)The Horse Race: TimescaleDB vs. MongoDBOnce again, we’ll use the common time-series scenario of DevOps metrics to benchmark the performance of the two MongoDB methods and TimescaleDB.Note:We've released all the code and data used for the benchmarks below as part of the open-sourceTime Series Benchmark Suite(TSBS).Machine configurationFor comparing both insert and read latency performance, we used the following setup:Version:TimescaleDB version 1.7.1community edition, with PostgreSQL 12,MongoDB 4.2.8 community edition. These were the latest production releases of both databases as of July 2020, at the time of testing.1 remote client machine, 1 database server, both in the same cloud data center.Instance size: both client and database server ran on DigitalOcean virtual machines (droplets) with 32 vCPU and 192 GB of memory each.OS: Both server and client machines ran Ubuntu 18.04.3Disk Size: 4 TB of remote SSD storage in a raid0 configuration (EXT4 filesystem), plus 800 GB of local SSD storage.Deployment method: database servers were deployed using Docker images, using images pulled from the official Docker hubs ofMongoDBandTimescale, respectively.Memory: both databases were given all available memory.Benchmarking datasetTo benchmark both write and read performance, we used the following dataset:Dataset: 4,000 simulated devices generated 10 CPU metrics every 10 seconds for 4 full days (~100M+ reading intervals, ~1B+ metrics)Dataset generated with theTime Series Benchmarking Suite, using thecpu-onlyuse caseDetails of the DevOps dataset used to benchmark both write (ingest) and read (query) performance for MongoDB and TimescaleDB.Batch size: Inserts were made using a batch size of 10,000Additional database configurations:For TimescaleDB, we set the chunk size to 12 hours, resulting in 6 total chunks (learn more about chunks and chunk time here).We also enablednative compression on TimescaleDB. We compressed all data older than 12 hours, resulting in 7 compressed chunks and 1 (data from the last 12 hours) uncompressed chunk. This configuration enables greater query efficiency and the ability to handle out-of-order data since raw data is kept for recent time periods and older data is compressed (see ourcompression docsfor more). Here are the compression parameters used: we segmented by thetags_idandhostnamecolumns and ordered bytimedescending andusage_usercolumns.Write Performance and Disk UsageInsert rate comparison: The simplicity of Mongo-naive led it to outperform Mongo-recommended by 54%, but TimescaleDB outperforms both methods - achieving 169% (vs. Mongo-naive) and 260% (Mongo-recommended) of the performance of MongoDB.Because NoSQL databases typically trade off some guarantees of relational databases, one might expect MongoDB to achieve better write performance/throughput, making it an inviting choice for ingesting time-series data, which can be at a rate of thousands of readings per second (or more).While it’s true that plain PostgreSQL does tend to lose write throughput as the dataset size grows (See our PostgreSQL vs. TimescaleDB benchmark resultsfor more), TimescaleDB’s unique chunking mechanism keeps write performance high (learn more aboutchunks).As a result,TimescaleDB outperformed both MongoDB configurations by a significant margin: TimescaleDB saw 169 % better insert performance compared to the Mongo-naive method and 260 % better write performance compared to the MongoDB-recommended.💡Note:We achieved these insert numbers using 32 concurrent clients inserting data into each setup. If you want to replicate this experiment, ensure your client machine has enough cores to execute this parallelism.Using parallel writes is a general best practice for improving write performance, as INSERTs and COPYs are generally executed as a single transaction and thus run in a single-threaded fashion (as is the case with PostgreSQL and TimescaleDB). For more guidance and details on achieving high ingest rates for time-series data, see our13 ways to improve TimescaleDB and PostgreSQL write performance.All three setups achieve write performance of over 1 million metrics per second. However, only TimescaleDB seems suitable for performance-critical, time-series use cases, as it achieved very high ingest rates on the high cardinality benchmark dataset, with an average insert rate of 2.7 million metrics per second. The sluggishness of the Mongo-recommended method’s ingest rate is likely due to the extra cost of occasionally creating new, larger documents (e.g. when a new hour or device is encountered).Given that time-series data piles up quickly and the costs associated with storing large amounts of data, it’s also worth exploring the disk storage footprint of each database configuration:Timescale uses 10x less disk space to store the same number of metrics than both MongoDB configurations.TimescaleDB uses 9 % of the disk space of the Mongo-naive and 8 % of the disk space of Mongo-recommended methods. This is thanks to the TimescaleDB’s novel hybrid row/columnar storage approach, which uses Gorilla compression for floats; delta-of-delta and simple-8b with run-length encoding for timestamps and integer-like types; whole-row dictionary compression for a few repeating values, with LZ compression on top; and LZ-based array compression for all other types.(Interested readers can learn more abouthow TimescaleDB’s native compression works, as well as thisexplanation of time-series compression algorithms and how they work.)Moreover, for the ~1 billion benchmark dataset, the Mongo-recommended method used more disk space than both the Mongo-naive method and TimescaleDB, making it worse than Mongo-naive on insert performance.That said,  we recommend doing an honest analysis of your insert needs. If your insert performance is far below these benchmarks (e.g., if it is 2,000 rows/second), then insert performance will not be your bottleneck, and this comparison becomes moot.Write Performance and Disk Usage SummaryTimescaleDB outperforms both methods of storing time-series data in MongoDB by between 69 % (vs. Mongo-naive) and 160% (vs. Mongo-recommended).Mongo-naive shows better write performance (154 % the ingest rate) and uses less disk space (85 % as much disk) than Mongo-recommended.TimescaleDB uses 10x less disk space than MongoDB, using 9 % of the storage footprint of Mongo-naive and 8 % of the storage footprint of Mongo-recommended.If your insert performance is far below these benchmarks (e.g., if it is 2,000 rows/second), then insert performance will not be your bottleneck.Query PerformancePart 1: Mongo-naive vs. Mongo-recommendedBefore we compared MongoDB against TimescaleDB, we first evaluated the query performance between the two MongoDB methods.By this point, Mongo-naive had demonstrated better write performance with simpler implementation and lower disk usage, but we suspected that Mongo-recommended would outperform Mongo-naive for query performance, justifying its recommendation by the MongoDB team and users.And, if there were a clear winner between the two methods for simple queries, we could save ourselves some time by not implementing our full query set against both methods.So, we first compared the two MongoDB methods using three “single rollup” queries (single groupby on time), one “double rollup” query (double groupby on time and device hostname), and one “aggregate” query (max reading over a time period). All queries were run with 1000 different parameter permutations (i.e., with random time ranges and hosts), from which we recorded the mean time for each database respectively.Latencies in this chart are all shown as milliseconds, with an additional column showing the relative performance of MongoDB-recommended compared to MongoDB-naive.Here are the results:Mongo-recommended outperforms Mongo-naive by 5x-68x, depending on the queryMongo-recommended outperforms Mongo-naive in all of the queries above, demonstrating 550-6,800 % of the performance of Mongo-naive. This performance difference makes it clear to see why it is the recommended method for storing time-series data in MongoDB, despite having slower ingest performance (1.5x slower) and more disk usage (16 % more) than Mongo-naive.Given this significant difference in query performance, compared to the modest difference in write performance and disk storage footprint, we decided that this approach was probably the best setup for storing time-series data in MongoDB.Thus, for the remainder of this post and our analysis, we use the “Mongo-recommended” setup whenever benchmarking MongoDB.(Note to readers: If all you want to get out of this post is the best way to store time-series data in MongoDB, here’s your answer: use the “Mongo-recommended” method. If ingest performance is more important to you than query performance, use “Mongo-naive.” But, if you want even better performance than either Mongo method, you may want to continue reading.)Part 2: TimescaleDB vs. MongoDBHaving settled on which MongoDB method is best, let’s evaluate MongoDB vs. TimescaleDB for querying time-series data.The best way to benchmark read latency is to do it with theactualqueries you plan to execute. For this case, we use a broad set of queries to mimic the most common query patterns. The results shown below are the average from 1000 queries with different parameter combinations (time range, device ID) for each query type.Latencies in this chart are all shown as milliseconds, with an additional column showing the relative performance of TimescaleDB compared to MongoDB (highlighted in orange when TimescaleDB is faster, in green when MongoDB is faster).Results of benchmarking query performance between MongoDB and TimescaleDB, with TimescaleDB showing 200 % to 5,400 % of the performance of MongoDBLet’s unpack the results for each query type below:Simple rollupsFor simple rollups (i.e., groupbys), when aggregating one metric across a single host for 1 or 12 hours or multiple metrics across one or multiple hosts (either for 1 hour or 12 hours), TimescaleDB performs comparably to or outperforms MongoDB.In particular, when aggregating one or more metrics on a single device for a single hour, the two databases show fairly equal performance. However, when aggregating one or more metrics across multiple devices for multiple hours,TimescaleDB shows between 208 % and 302 % of the performance of MongoDB.AggregatesFor calculating a simple aggregate (i.e., finding the maximum value) for metrics from one or more devices, TimescaleDB outperforms MongoDB. In our benchmark,TimescaleDB demonstrates 396 % of the performance of MongoDB when aggregating 8 metrics across 4,000 devices and 195 % when aggregating one metric across 4,000 devices.Double rollupsWhile single rollups and aggregates are somewhat comparable across the two databases, other more complex queries are not. For double rollups aggregating metrics by time and another dimension (e.g., GROUPBY time, deviceId), TimescaleDB shows large gains.When aggregating one metric per device, per hour, for some 24-hour window, TimescaleDB showed 1,507 % (or 15x) the performance of MongoDB.When aggregating 10 metrics, Timescale showed 1,327 % (or 13x) of the performance of MongoDB.The largest performance difference came when aggregating 5 metrics per device, per hour, for some 24-hour window, where TimescaleDB achieved 2,149 % the performance of MongoDB, or 21x.Notice that these queries take in the order of tens of seconds (rather than milliseconds), so a 13-21x performance gain is very noticeable.Complex queriesFor complex queries that go beyond simple rollups or aggregates, the comparison is much more clear-cut: TimescaleDBvastlyoutperforms MongoDB here (in some cases, more than 50 times faster).LastpointThe first complex query (‘lastpoint’) finds the latest reading for every device in the dataset. This query typeis commonly used in IoT and DevOpsfor analysis and monitoring.While our dataset has all devices reporting at consistent intervals, this query can be troublesome to implement in the general case because it could be that some devices have not reported in quite a long time, potentially causing a lot of documents (MongoDB) or rows (TimescaleDB) to be scanned.We are able to do some clever query construction in both to get a list of distinct devices, which allows both setups to stop searching when every device has a point associated with it.We utilize JOINs in both systems. However, while MongoDB does support JOINs, they are not as natural to work with or ""feature-full"" as they are for relational databases like TimescaleDB.For the more complex lastpoint query, TimescaleDB shows 5,399 % (or 54x) of the performance of MongoDB.Groupby-Orderby-LimitThe second complex query (‘groupby-orderby-limit’) does a single rollup on time to get the MAX reading of a CPU metric on a per-minute basis for the last 5 intervals for which there are readings before a specified end time. This is another type of query common in DevOps and IT monitoring workloads.This is tricky because those last 5 intervals could be the 5 minutes before the end time, or if there is no data for some minute periods (i.e., “gaps”), they could be spread out, potentially needing a search from the beginning just to find all 5.In fact, a full table scan was the query strategy needed for MongoDB, while TimescaleDB has intelligent planning to utilize its indexes.This resulted in TimescaleDB showing 2,108 % (or 21x) of the performance of MongoDB.Read performance summary:For simple queries and aggregates, TimescaleDB generally outperforms MongoDB, showing between 1-4x better performance than MongoDB.For double rollups, TimescaleDB vastly outperforms MongoDB, showing 13-21x performance improvement.For complex queries, which are commonly used to analyze and monitor devices for DevOps and IoT use cases, TimescaleDB again vastly outperforms MongoDB, showing up to 53x better performance.If query performance is your most important requirement, TimescaleDB is the clear choice for both simple and complex queries.NoSQL vs. SQL: Query Language ComparisonPerformance is not the only aspect to consider when deciding which database to use to store time-series data. Developer experience and implementation complexity are also important factors.With that in mind, we compared the query language differences between TimescaleDB and MongoDB, using the complex query “groupby-orderby-limit” from our performance analysisOnce again, we may be biased, but we found SQL —with 5 lines of code—much simpler than MongoDB's document format—with 72 lines of code. This is a crucial criterion for sustainable software development since humans create, maintain, and use these systems at the end of the day. This is another point in favor of SQL in the ongoingNoSQL vs. SQLdebate.TimescaleDB query (SQL):SELECT date_trunc('minute', time) AS minute, max(usage_user)
FROM cpu
WHERE time < '2016-01-01 19:47:52.646325 -7:00'
GROUP BY minute
ORDER BY minute DESC
LIMIT 5And here’s that same query expressed in MongoDB.MongoDB query:[
  {
    $match: {
      measurement: ""cpu"",
      key_id: {
        $in: [""20160101_00"", ""20160101_01"", ""20160101_02"", ""20160101_03"", ""20160101_04"", ""20160101_05"", ""20160101_06"", ""20160101_07"", ""20160101_08"", ""20160101_09"", ""20160101_10"", ""20160101_11"", ""20160101_12"", ""20160101_13"", ""20160101_14"", ""20160101_15"", ""20160101_16"", ""20160101_17"", ""20160101_18"", ""20160101_19"", ""20160101_20"", ""20160101_21"", ""20160101_22"", ""20160101_23"", ""20160102_00"", ""20160102_01"", ""20160102_02"", ""20160102_03"", ""20160102_04"", ""20160102_05"", ""20160102_06"", ""20160102_07"", ""20160102_08"", ""20160102_09"", ""20160102_10"", ""20160102_11"", ""20160102_12"", ""20160102_13"", ""20160102_14"", ""20160102_15"", ""20160102_16"", ""20160102_17"", ""20160102_18"", ""20160102_19"", ""20160102_20"", ""20160102_21"", ""20160102_22"", ""20160102_23"", ""20160103_00"", ""20160103_01"", ""20160103_02"", ""20160103_03"", ""20160103_04"", ""20160103_05"", ""20160103_06"", ""20160103_07"", ""20160103_08"", ""20160103_09"", ""20160103_10"", ""20160103_11"", ""20160103_12"", ""20160103_13""]
      }
    }
  },
  {
    $project: {
      _id: 0,
      key_id: 1,
      tags: ""$tags.hostname"",
      events: 1
    }
  },
  {$unwind: ""$events""},
  {
    $project: {
      key_id: 1,
      tags: 1,
      events: {
        $filter: {
          input: ""$events"",
          as: ""event"",
          cond: {
            $and: [
              {$gte: [""$$event.timestamp_ns"", 1451606400000000000]},
              {$lt: [""$$event.timestamp_ns"", 1451827606646325489]}
            ]
          }
        }
      }
    }
  },
  {$unwind:$events},
  {
    $project: {
      time_bucket: {
        $subtract: [
          ""$events.timestamp_ns"",
          {$mod: [""$events.timestamp_ns"", 60000000000]}
        ]
      },
      field: ""$events.usage_user""
    }
  },
  {
    $group: {
      _id: ""$time_bucket"",
      max_value: {$max: ""$field""}
    }
  },
  {$sort: {_id: -1}},
  {$limit: 5}
]Two things that jump out besides the verbose JSON syntax:First, to efficiently stop the query, the client running the query will have to compute the subset of documents to look in, which creates the lengthy list in the first $match aggregator above.Second, to unpack the 60x60 matrices in each document, the $unwind/$project/$unwind pattern is needed to efficiently expand those matrices while removing empty time periods. This pattern is actually needed for almost all of the queries we looked at here, which makes all the queries verbose and potentially daunting to debug.In this example, the SQL query is far shorter and simpler, making it easier to comprehend and debug. This is not the only advantage of using SQL (and TimescaleDB) to query time-series data: SQL has a rich tradition and history, including familiarity among millions of developers and a vibrant ecosystem of tutorials, training, and community leaders.In short, as thethird-most commonly used programming language among developers, choosing SQL means you’re never alone.Most popular Programming, Scripting, and Markup Languages.Source:2020 Stack Overflow Developer SurveyConclusionTimescaleDB delivers 260 % higher insert performance, up to 53x faster queries and better developer experience vs. MongoDBIs it surprising that TimescaleDB, a database purpose-built for time-series data, outperforms MongoDB, a general-purpose document store when it comes to time-series data?Not necessarily—but there are enough blogs, talks, and other material out there about using MongoDB for time-series data that we felt we needed to do an evaluation.We understand that for many users, MongoDB offers the benefit of being easy to learn and quick to set up. Yet, for time-series data, setting up MongoDB to deliver the performance your workloads require is not simple, as we’ve shown in our analysis, and requires careful thought about your schema design.If you simply dump each reading into a new document, you’re in for longer and longer wait times as more data accumulates and you want to query it. Moreover, implementing MongoDB's recommended time-series approach requires writing client-side code and using fairly verbose queries.We may be biased, but we'd contend that rather than doing all of the above and forcing MongoDB—a general-purpose document store—to behave like a time-series database, opting for a database built specifically for time-series data is a better use of time and resources.And, as we've shown, when it comes to time-series workloads, TimescaleDBa purpose-built time-series database—delivers significantly better results on every dimension. If you're weighing your options, based on our analysis, TimescaleDB is the clear choice.P.S. If you’re looking for more comparisons of database performance for time-series data, read our studies comparingAmazon Timestream vs. TimescaleDBandInfluxDB vs. TimescaleDB.Want to Learn More About TimescaleDB?Create a free accountto get started with a fully managed TimescaleDB instance (100 % free for 30 days).Want to host TimescaleDB yourself?Visit our GitHubto learn more about options, get installation instructions, and more (and, as always, ⭐️  are  appreciated!)Join ourSlack communityto ask questions, get advice, and connect with other developers (our co-founders, engineers, and passionate community members are active on all channels).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-store-time-series-data-mongodb-vs-timescaledb-postgresql-a73939734016/
2019-08-28T17:08:01.000Z,How to Benchmark IoT Time-Series Workloads in a Production Environment,"New IoT dataset available for benchmarking TimescaleDB, MongoDB, InfluxDB, Cassandra & others.Benchmarking ain’t easy. That is true in general, but can be particularly challenging when comparing the performance of various database technologies for specific workloads. And although it is not the simplest task, it is extremely important to test out these technologies before running them in production scenarios.Here, our focus is on the IoT industry where we’ve found that many organizations often collect large amounts of  time-series data as part of their operations. By collecting data in this way, they are able to conduct real-time and even predictive analysis to better inform current and future decision making.Now that we understand the value of collecting time-series data, the next logical question becomes: what database should we deploy for an IoT use case? We built a tool to help you answer that question.Enter the Time Series Benchmark SuiteLast year weannouncedthe availability of theTime Series Benchmark Suite (TSBS)as a tool for comparing and evaluating databases for time-series workloads. The primary goal of TSBS is help take the guesswork out of benchmarking by simplifying the experience of generating time-series datasets, and comparing insert & query performance on a host of database technologies.Currently TSBS supports TimescaleDB, MongoDB, InfluxDB, Cassandra, ClickHouse, CrateDB and SiriDB. TSBS is designed to be transparent with benchmarking, allowing others to compare and run their own results.(If there’s a technology you’d like to see added, you can submit a GitHubpull request.)The key with TSBS is that we include specific time-series use cases to bring context to your testing and mimic the workload in a production scenario. We initially released a suite of tests with TSBS that was designed to measure the performance of these database technologies when implemented in the DevOps use case.Over the past year, we spoke with a number of users who are incorporating time-series workloads into their IoT architecture and the topic of benchmarking was always on the agenda. We took a step back and decided to help users address this topic by expanding the TSBS to include an IoT-specific use case. This use case includes the ability to:1. Generate data sets that simulate an IoT scenario2. Benchmark insert/write performance to simulate loading IoT data into database3. Benchmark query execution performance to simulate analyzing IoT data in the databaseIn this post, we will share our reasoning behind adding the IoT use case by showcasing how it can be used to simulate an end-to-end real-world IoT scenario. We will also share benchmarking results produced using TSBS for this IoT use case against two popular time-series databases (TimescaleDB and InfluxDB).If you’ve heard enough and you are interested in trying it out yourself, go straight to theGitHub page. Or see examples of the results TSBS can producehereandhere.Otherwise, please continue reading.How to generate and load IoT dataAs mentioned above, using TSBS for benchmarking involves these phases: generating data sets; benchmarking insert (i.e. “loading”) performance; and benchmarking query (i.e. “analysis”) performance.For this new IoT use case, we envisioned these phases playing out based on a fictional trucking company. To demonstrate, we will showcase how to profile data generation by tracking a series of metrics across a fleet of trucks, and offer queries for both real-time analysis of truck activity and predictive analysis for truck maintenance.In this scenario, there is a device attached to the truck which observes and reports the truck data that we need. The data is a combination of location, diagnostics, and metadata. And as in the real world, data sets are never “perfect” so TSBS also simulates real events that cause the data set to be “less than perfect”.(LOCATION + DIAGNOSTICS + METADATA) & (REAL EVENTS) = “The IoT Data Set”Calculating location, diagnostic, and metadataThe location of the truck (as measured by the GPS unit in the form of longitude and latitude of the truck) would be combined with the heading as reported by the sending unit. All of these measurements are reported as individual bits of data on a time interval. This would make up the data that we need to be able to track the truck in real-time (last reported longitude, latitude, and heading). Additionally, we can follow the route of the truck and map all of the historical data points as they relate to positioning and heading from that truck.(Truck location) metrics we can collect:TimeLongitudeLatitudeElevationVelocityHeadingGradeIn addition to collecting (truck location) metrics, we also want to be able to analyze the current conditions (or diagnostics) of the truck as it travels overtime. For example, is the truck low on fuel, or when was the last time the truck stopped for a period of time?(Truck) diagnostics we can collect:TimeFuel stateFuel consumptionCurrent loadTruck status (moving or stationary)Next, we would want to introduce (truck) metadata, which allows us to tie our queries to more static assets. For example, what is the average stop time (out of service time) for a given truck model in the fleet? Which drivers obey the speed limits and which present risk? When we start to tie our variable data (metrics) to static metadata, we can start to perform analytics as we roll the data up under these bits of metadata. This is another type of query in the benchmarking stack.(Truck) metadata we can collect:FleetDriverTruck modelDevice versionLoad capacityFuel capacityNominal fuel consumptionSimulating real eventsNow that we have looked at the data that can be generated, we need to actually create the data sets that also simulate “real world” events that could have an impact on how the system may perform.Let’s assume that the trucks are transmitting via a wireless network, however, out of network trucks will queue the data locally and when connectivity is established it will push a batch update. We would want to know how this impacts system performance if it happens to a large number of trucks.The items below represent some of the real events that could impact the system, which we account for in our benchmarking runs.List of real events that can affect data collection:Lots of gaps in the data (representing trucks whose reporting device is broken)Out of order ingestion (trucks out of network range, and come back in and report batched data)Different versions of input (not all devices on the trucks will run the same firmware)Need to correlate data between different time-seriesOften contains geospatial informationHow to perform a real-time analysis on the IoT dataFrom above, we have generated a data set that represents a real IoT use case (and this data set is not “perfect” because we’ve injected real events). Using this data set, we are ready to benchmark insert (“loading”) performance and query (“analysis”) performance. Let's spend some time walking through a preliminary set of benchmark results when we run this againstTimescaleDB, a time-series database on PostgreSQL.Setting up your test platformWe will start by defining the test system. In this case we are running the tests against a system with the following specs:8 CPU30GB RAM1TB Storage (SSD)In this test we used 2 sets of configurations:One set of queries were run 500 times using 8 workersAnother set of queries was run 100 times using 4 workersNote: We indicate at the beginning of each result section the frequency and numbers of workers used.The other set of variables you will need to understand is the size and scale of the data set. In this case we used the following:Size of data set: 3 months worth of data (38 million rows) based on data described aboveScale of data set: 1000 trucksNote: These parameters are set when you run the script that generates your data, for more information on this, please see the TSBS GitHub repohere.Measuring ingestion ratesThe first measurable statistic we want to look at in our run is how did the database perform during ingestion. Here we are looking at how quickly we were able to take the data from a generated file, introduce our real world scenarios, and see how the database performs.TimescaleDB ingestion rates (rows per second):We see that the loading of the dataset, which included variability introduced by the “real world” events we discussed previously, resulted in TimescaleDB loading the generated 38.8 million rows at a rate of 369,141.6 rows per second. This means we were able to load the data set in 105.3 seconds, or in less than 2 minutes.Another way to look at this (as a way or normalizing results given the different data models used by various time series databases that TSBS supports) is to view at this in terms of number of metrics per second.TimescaleDB ingestion rates (metrics per second):When we normalize the data to metrics per second, we see that we are loading 1.8 million metrics per second which gets us to the same total load time. However, we also get a metric to compare against different data models where “rows” is not a relevant concept.Now that we have looked at some of the basic ingestion numbers and have a good sense for how our database has performed in loading the data, let's take a look at how some of our queries performed.Testing against sample queriesBefore we jump in, we will start with a brief description of each query.To start producing results, we will walk through a single query and explain how to interpret the results. Next we will broaden the result set so you can see how all of the queries performed.Let's start by looking at a query we ran that will show us every truck’s last location (our data set here consists of 1,000 trucks).Total average for a single query in TimescaleDB that we ran 500 times using 8 workers:Here we queried the database for the last known location of each truck, and on average the database was able to return our result set within 151.64 milliseconds. Given we ran this query 500 times, it is also important to look at the spread of the results. In this case we had a standard deviation of 21.17 ms, meaning on average our results were +/- 21.17 ms from the average query completion time (21.17 ms making up one standard deviation). Expressed another way, our completion time per query was 14% different from the average query completion time. This represents the spread or variability of the results in our run: the higher this number, the more variable our results.Now that we have walked through the results for a single query, let’s present the rest of results where we run through multiple queries using a larger and smaller configuration.Multiple queries in TimescaleDB that we ran 500 times using 8 workers:Multiple queries in TimescaleDB that we ran 100 times using 4 workers:As you can see, we are presenting the results in such a way that you can get a good sense of what you should expect not just based on TimescaleDB, but also considering the scale of the machine that we have chosen to run on, which will also impact the level of performance. We are also mixing sets of simple and complex queries to complete the profile of what you can expect.Benchmarks: TimescaleDB vs. InfluxDBA big part of benchmarking is to have an idea about how your system will perform. The other large piece of benchmarking is trying to perform an “apples-to-apples” comparison of two systems. The above covered how TimescaleDB performed executing an IoT use case. For reference, let's look at a preliminary set of query performance metrics for an alternative time-series database such as InfluxDB.We start with the same set of data ingestion tests, and measure how InfluxDB performed as it loaded our dataset into the database.InfluxDB ingestion rates (rows per second):In this run, we see that the data load took a little more than 3 minutes at 170,045 rows per second. If we want to normalize the data to metrics loaded per second, we can look at the ingestion numbers from that perspective as well.InfluxDB ingestion rates (metrics per second):You may notice the difference in the number of metrics. This is one factor that will differ from platform to platform (and why we have a separate data generation process per platform). Here we see that InfluxDB loaded data at a rate of 1.3 million metrics per second.Next, let’s take a look at how InfluxDB performs using the same queries as the previous test against TimescaleDB.Multiple queries in InfluxDB that we that we ran 500 times using 8 workers:Multiple queries in InfluxDB that we ran 100 times using 4 workers:As you can see, TSBS is able to show us the benchmark results around system performance based on the size and shape of the platform that it is running on (CPU, Memorey, Disk), and the data platform used to run the tests. Here, the common thread is the same data and query patterns based on a real world use case.In the above, we chose to use TSBS against both TimescaleDB and InfluxDB to run our tests, and were able to produce two independent, but comparable, sets of results.Comparison of TimescaleDB vs. InfluxDB of the 100 run / 4 worker benchmark results:Next stepsThe idea behind TSBS is to exercise the database in such a way that it will reflect real world IoT use cases of the data and how the database will respond. As a result, we can achieve realistic performance numbers based on a real production workload.This newly added set of benchmarks are very specific to the IoT use case, and account for the different variables that these workloads encounter in a production environment. The end result allows users to benchmark their system against something that has all of the characteristics of what you will see in production, and really get an understanding of what you can expect in that production environment.If you are ready to try out TSBS for your IoT use case, visitherefor instructions.Contributions and suggestionsFinally, we’d like to emphasize that TSBS is an open source project, welcoming contributions from the community. If you have feedback for us or ideas around the next set of use cases you would like to see us address, please feel free to swing by theGitHub repo and open an issuewith your ideas.Additionally, if you are interested in benchmarking against databases outside of what is currently supported, your contributions and suggestions are welcome. Simply open an issue, or a pull requestvia the repository.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-benchmark-iot-time-series-workloads-in-a-production-environment/
2023-09-01T20:18:00.000Z,Best Practices for Query Performance on PostgreSQL,"In aprevious blog post, we shared for the first time how we are building a distributed time-series database on PostgreSQL by relying on chunking instead of sharding. To that end, we also introduced distributed hypertables and showed how they could scale almost linearly for inserts up to a nine-node cluster (one access node and eight data nodes), with an insert rate of well over 12 million metrics a second.In this post, we take a look at how we optimize queries on distributed hypertables in our multi-node architecture and share some query benchmarks. We've seen significant performance benefits with distributed hypertables over regular hypertables for both full and partial aggregations.But first, let's look at the main causes of performance degradation on PostgreSQL.What Leads to Performance Problems?Performance issues can significantly impact the efficiency and functionality of a PostgreSQL database. Several factors can contribute to these problems. Here are some of the main causes:Inefficient Queries:One of the most common causes of performance problems is inefficient queries. Poorly optimized or complex queries can create bottlenecks and slow down database operations. Regularly reviewing and optimizing queries can help mitigate this issue.Insufficientindexes:A lack of appropriate indexes can lead to slower query execution as the database must scan entire tables to find relevant data.Implementing indexes on frequently queried columnscan significantly improve query performance.ImproperPostgreSQL datatypes:Using inappropriate data types can also impact performance. For example, using a text data type when an integer would suffice leads to unnecessary storage usage, potential casting overhead, and slower query speeds.Changing data volume:The query planner creates execution plans (how to run a query) based on statistics of source data. Increasing or decreasing the data volume can influence these stats and yield different execution plans, resulting in slow queries that were once fine. Analyzing the source table can help optimize the query for a different volume.Highvolume oftransactions:A high volume of transactions can strain system resources and lead to performance degradation. Implementing efficient transaction handling strategies can help manage this load.Insufficienthardwareresources:Hardware limitations, such as CPU, memory, or storage capacity, can hinder database performance. Upgrading hardware resources or optimizing resource allocation can often resolve these issues.Lock contention:Using locks on tables or rowscan slow down a system with parallel queries, basically synchronizing access to the data and creating waiting queues for requests. Locks should be used very carefully and only as a last result.Lack ofmaintenance:Routine maintenance tasks like vacuuming, reindexing, and updating statistics are crucial for maintaining optimal performance. Neglecting these tasks can lead to performance degradation over time.By addressing these common causes of performance issues, you can optimize your PostgreSQL database for better performance and responsiveness.Let's now jump straight into our benchmarks.Query Performance BenchmarksTo show the query performance of distributed hypertables, we use theTime-Series Benchmark Suite (TSBS)with the IT monitoring use case (DevOps). All of the benchmark results are produced on nodes that ran m5.2xlarge AWS instances.For the IT monitoring use case, we have a distributed hypertable with timestamped CPU metrics that cover multiple hosts being monitored. The distributed hypertable is partitioned along two dimensions:timeandhostname, where the hostname determines which data node a chunk is placed on.In other words, each data node stores data for the same time intervals but covers a specific subset of hosts, as shown below.Let’s first look at the query performance of computing the average hourly CPU usage per host.With eight data nodes, queries complete almost 8x faster than with one data node on a distributed hypertable.We include one data node as a point of reference to show the slight overhead of distributed hypertables due to the extra network communication and processing. Even with that slight overhead, we see that queries complete about 7x faster on a distributed hypertable with eight data nodes than on a regular (one node) hypertable.The reason this query performs well is because it can push most of the work down to the data nodes.Since the query is grouping based on hostname, each data node can compute afull aggregateindependently and concurrently (more on that later).But what if we query for something less ideal for this setup? For instance, let’s look at finding the max CPU usage per hour across a set of hosts. Such a query groups only on time, and data nodes can therefore only computepartial aggregates. To get the final result, the access node has to combine and finalize the partial results from the data nodes.Even in this case, eight data nodes are over 3x faster than a single data node.More notably, the overhead between hypertables and distributed hypertables is much smaller here than in the prior benchmark, bringing the two roughly on par. This is because the number of rows returned is much smaller for this query than the previous one, where we also grouped by hostname.How We Optimize Queries on Distributed Hypertables (With an Example)The key to unlocking the full query performance of distributed hypertables boils down to three main tactics:Limiting the amount of workOptimally distributing and pushing down work to data nodesKeeping data nodes busyWe’ll now discuss how Timescale incorporates each of these tactics.Limiting the amount of workOne of the first things that the access node does with a query is to determine which chunks are involved in the query and, by extension, which data nodes to talk to. For instance, let’s say we are executing the following query, which is similar to the one in the first benchmark test:SELECT time_bucket(time, ‘1 hour’) as hour, 
	hostname, avg(cpu) 
FROM measurements 
WHERE hostname IN (‘host001’, ‘host002’) 
	AND time > NOW() - interval ‘24 hours’
GROUP BY hour, hostname;This query tells us that we only want to work with the set of chunks whose time intervals overlap with the past 24 hours and include, e.g., hostnameshost001and/orhost002.Even though the access node doesn’t store any data locally, it has global knowledge of all chunks across the distributed hypertable, allowing it to performchunk exclusion. Each chunk in the resulting set has a list of data nodes that store a copy of it, allowing the access node to turn the set of chunks into a set of data nodes to be queried, as shown in the figure below.Note that chunks may have replica copies on multiple data nodes for fault-tolerance purposes; in that case, the access node has a choice of which data nodes to query for optimal performance.Different strategies for assigning chunks are possible, but the default strategy is always to use the designated “default” data node for a chunk.At the end of this planning stage, the access node has a list of data nodes to query, each with a disjoint subset of chunks. From this, it synthesizes a SQL query to send to each data node based on the original query:SELECT time_bucket(time, ‘1 hour’) as hour, 
	hostname, avg(cpu) 
FROM measurements 
WHERE _timescaledb_internal.chunks_in(measurement, ARRAY[1, 2])
hostname IN (‘host001’, ‘host002’) 
	AND time > NOW() - interval ‘24 hours’
GROUP BY hour, hostname;Note that the access node explicitly tells the data node which chunks to query via thechunks_infunction in theWHEREclause. This function serves two purposes: first, it obviates the need for running chunk exclusion again on the data node, and second, it avoids returning duplicate data from replica chunks that also exist on other data nodes.Thus, at the end of this planning stage, the access node knows exactly which data nodes to talk to and which chunks to query on those nodes.Pushing Down Work to Data NodesWhen generating the SQL query statement to send to a particular data node, the access node needs to decide which parts of the original query it can safely push down, i.e., execute on the data nodes, and which parts need to execute locally.In the worst case (i.e., no push down), the access node has to fetch the raw data from each data node and process it locally, as shown below:Clearly, fetching the raw data is to be avoided if possible, as it involves transferring a lot of data and puts a heavy processing burden on the access node. Instead, push-downs have the potential to improve the situation tremendously by (1) moving processing to the data nodes and (2) reducing the amount of transferred data since the output is often smaller than the input, especially in the case of aggregations and limits.Typical things the planner can consider to push down include:Functions (and general expressions)Sorting (ORDER BY)LIMITsAggregates and GROUP BYsWhile a number of things determine the ability to push down various parts of the query, we will focus on the ability to push down the computation of GROUP BY aggregates (e.g., calculating the average CPU usage per host and hour).Full aggregationGoing back to our example query, the first thing that determines the level of aggregate push-down is whether the GROUP BY clause covers all of the partitioning dimensions. Because the GROUP BY clause includes both time and hostname (both of which are partitioning dimensions), we know that it is safe to push down the aggregation fully.This is becauseno data for a time-hostname group on a data node can exist on any other data node. Thus, with full aggregate push-down, the query conceptually looks as follows:Note, however, that we aren’t actually grouping on time and hostname; we are grouping onhour(time bucket) and hostname. Fortunately, since we use hostname as our primary dimension along which we assign chunks to data nodes, the planner can think of this as a single-dimensional table partitioned only on hostname.This allows it to do time bucketing independently on each data node safely. There’s a caveat here, though: repartitioning. If the table has been repartitioned, data might not align along data node boundaries as we require, so full aggregation might not be possible (more on that below).In summary, full aggregation is possible if any of the following cases hold:The grouping clause includes all partitioning keysThe grouping clause includes only the “space” partitioning key, and the time restriction includes no repartitioning eventThe query is restricted to only one data nodeFortunately, the TimescaleDB planner is smart enough to detect and handle each case for optimal query performance.Partial aggregationPartial aggregation is necessary when the data for a computed group is not located on a single data node [for example, when we group only by hour (time)]. In this case, each data node computes a partial aggregate, and the access node then finalizes the result for each hour bucket.Conceptually, this looks as follows:While PostgreSQL fully supports partial aggregations on a local node (e.g., for parallel query execution), there is unfortunately no general way to express a partial aggregation in standard SQL, which is necessary to tell a data node that it should compute a partial aggregate.In the case ofavg(), this partial aggregate state would consist of a sum and a count that can be used to produce the final average. But, obviously, this state is different depending on the aggregate function. Fortunately, TimescaleDB has a function for computing the partial aggregate state that is also used bycontinuous aggregates.The SQL sent to a data node would then look something like this:SELECT time_bucket(time, ‘1 hour’) as hour, 
	_timescaledb_internal.partialize_agg(avg(cpu))
FROM measurements 
WHERE _timescaledb_internal.chunks_in(measurement, ARRAY[1, 2])
hostname IN (‘host001’, ‘host002’) 
	AND time > NOW() - interval ‘24 hours’
GROUP BY hour;Note the addition of thepartialize_aggfunction around the aggregate. This function tells the data node to compute and return the partial state for theavg()aggregate so that the access node can compute the final aggregate from all the data nodes' partial results.How repartitioning affects push-downIt is easy to expand the capacity of a distributed hypertable by adding additional data nodes. But to use the extra data nodes, existing distributed hypertables might require repartitioning to, e.g., increase the number of space partitions.However, since the new partitioning configuration only affects new chunks, the planner has to be careful to ensure that queries on data nodes still produce the correct results. To illustrate this situation, let’s look at what happens when we add a new data node to expand the capacity of a distributed hypertable:The figure shows that, during the third time interval, an extra data node was added so that the fourth time interval now includes four chunks instead of three. Now, imagine that the highlighted area shows the chunks covered by a query’s time and hostname restrictions.We find that this includesoverlappingchunks from two distinct partitioning configurations, i.e., data for a particular hostname might exist on more than one data node. This will prohibit full aggregations on data nodes, as it would otherwise produce the wrong result.Fortunately, TimescaleDB's planner can dynamically detect overlapping chunks and revert to the appropriate partial aggregation plan when necessary. Users can, therefore, freely add data nodes and repartition their data to achieve elasticity without worrying about the correctness of query results.While this leads to slightly worse query plans in some cases, these repartitioning events are rare and often quickly move out of the query or retention window. There’s also the possibility of rewriting old chunks in the new partitioning scheme, although no automation for such repartitioning of old data currently exists.Keeping data nodes busyTo minimize latency and maximize resource utilization, it is crucial that the access node keeps feeding data nodes with work. Unfortunately, the default behavior of PostgreSQL is to execute so-called Append plans serially, i.e., the access node starts with getting all tuples from the first data node, then moves on to the second, and so forth.This is obviously bad for performance and should be avoided. PostgreSQL 11 introduced parallel append plans, but they require launching separate worker processes on the access node, which is a lot of overhead when most of the work anyhow happens on data nodes. Further, parallel workers introduce other challenges related to read consistency and coordination of two-phase commit across multiple connections to the same data node.Instead of using parallel append, TimescaleDB introduces asynchronous append execution that allows the access node to asynchronously initiate the query work on each data node while fetching the results as they become ready.Basically, like the event-driven paradigm, the idea is to eliminate as much idle time as possible across both the access node and all data nodes. This provides great performance improvements for push-down aggregates since most of the work happens simultaneously on data nodes.Best Practices for Query PerformanceHere are some general practices that can further improve your query performance:Understand common performance bottlenecks:know the usual suspects of performance issues, such as inefficient queries, missing indexes, or hardware limitations.Utilize chunk exclusion:use TimescaleDB's chunk exclusionto optimize the selection of relevant data chunks for faster query processing.Implement full aggregate push-down:maximize the push-down of aggregation to data nodes when theGROUP BYclause covers all partitioning dimensions.Employ partial aggregation for complex queries:use partial aggregations when data for a computed group is spread across multiple data nodes.Monitor repartition effects on push-down:be aware of how repartitioning, especially with added data nodes, affects the optimization of push-down operations.Asynchronously execute appends:adopt asynchronous append execution to minimize idle time and initiate work on each data node concurrently.Continuously update and educate:regularly update your knowledge, tools,and PostgreSQL versionto benefit from performance improvements and stay informed on best practices.Next StepsIf you want to learn more about PostgreSQL performance issues, including how to identify and solve them, plus how to optimize your query performance, read the following resources:Using pg_stat_statements to Optimize QueriesPostgreSQL + TimescaleDB: 1,000x Faster Queries, 90 % Data Compression, and Much More13 Tips to Improve PostgreSQL Insert PerformanceTo try distributed hypertables, you canself-host TimescaleDBorsign up for a free 30-day trial of Timescale, our cloud solution, which includes our core database but supercharges it with features designed for higher performance, faster queries, and less spending.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/achieving-optimal-query-performance-with-a-distributed-time-series-database-on-postgresql/
2023-07-18T15:00:00.000Z,TimescaleDB vs. InfluxDB: Purpose Built Differently for Time-Series Data,"📝 This article was initially published in August 2018 and updated in July 2023—this is, before Influx 3.0.Read this most recent article to get our thoughts on InfluxDB 3.0.Time-series datais emerging in more and more applications, including IoT, DevOps, Finance, Retail, Logistics, Oil and Gas, Manufacturing, Automotive, Aerospace, SaaS, even machine learning and AI. In the past, the focus of time-series databases has been narrowly on metrics and monitoring; today, it’s become clear that software developers really need a true time-series database designed for a variety of operational workloads.If you are investing in a time-series database, that likely means you already have a meaningful amount oftime-series datapiling up quickly and need a place to store and analyze it. You may also recognize that the survival of your business will depend on the database you choose.How to Choose a Time-Series DatabaseThere are several factors to consider when evaluating a time-series database for your workloads:Data modelQuery languageReliabilityPerformanceEcosystemOperational managementCompany and community supportIn this article, we compare two leading time-series databases,TimescaleDBandInfluxDB, to help software developers choose the time-series database best suited for their needs.Typically database comparisons focus on performance benchmarks. Yet performance is just a part of the overall picture. It doesn't matter how well a database performs in benchmarks if it lacks the data model, query language, or reliability required for your production workloads. With that in mind, we begin by comparing TimescaleDB and InfluxDB across three qualitative dimensions,data model, query language, and reliability,before diving deeper withperformance benchmarks. We then round out with a comparison withdatabase ecosystem, operational management, and company/community support.Yes, we are the developers of TimescaleDB, so you might quickly disregard our comparison as biased. But if you let the analysis speak for itself, you’ll find that we stay objective .Also, this comparison isn’t a purely theoretical activity for us. Our company began as an IoT platform, where we first used InfluxDB to store our sensor data. However, owing to most of the differences listed below, we found InfluxDB unsatisfactory. So, we built TimescaleDB as the first time-series database that satisfied our needs, and then discovered others who needed it as well, which is when we decided to open source the database.Today, just over three years later, the TimescaleDB developer community has come a long way, with tens of millions of downloads and over 500,000 active databases all over the world. This community includes organizations like AppDynamics, Bosch, Cisco, Comcast, Fujitsu, IBM, Schneider Electric, Samsung, Siemens, Uber, Warner Music, and thousands of others.In the end, our goal is to help you decide which is the best time-series database for your needs.Data modelDatabases are opinionated. The way a database chooses to model and store your data determines what you can do with it.When it comes to data models, TimescaleDB and InfluxDB have two very different opinions: TimescaleDB is a relational database, while InfluxDB is more of a custom, NoSQL, non-relational database. What this means is that TimescaleDB relies on therelational data model, commonly found in PostgreSQL, MySQL, SQL Server, Oracle, etc. On the other hand, InfluxDB has developed its own custom data model, which, for the purpose of this comparison, we’ll call thetagset data model.Relational data modelTherelational data modelhas been in use for several decades now. With the relational model in TimescaleDB, each time-series measurement is recorded in its own row, with a time field followed by any number of other fields, which can be floats, ints, strings, booleans, arrays, JSON blobs,geospatial dimensions, date/time/timestamps, currencies, binary data, or evenmore complex data types. One can create indexes on any one field (standard indexes) or multiple fields (composite indexes), or on expressions like functions, or even limit an index to a subset of rows (partial index). Any of these fields can be used as aforeign keyto secondary tables, which can then store additional metadata.An example is below:An example relational data model for time-series dataThe advantage of this approach is that it is quite flexible. One can choose to have:A narrow or wide table, depending on how much data and metadata to record per readingMany indexes to speed up queries or few indexes to reduce disk usageDenormalized metadata within the measurement row, or normalized metadata that lives in a separate table, either of which can be updated at any time (although it is easier to update in the latter case)A rigid schema that validates input types or a schemaless JSON blob to increase iteration speedCheck constraints that validate inputs, for example checking for uniqueness or non-null valuesThe disadvantage of this approach is that to get started, one needs to generally choose a schema, and explicitly decide whether or not to have indexes.Note: In the past several years, it’s been popular to criticize the relational model by claiming that it is not scalable. However,as we have already shown, this is simply not true: relational databases can indeed scale very well for time-series data.Tagset data modelWith the InfluxDBtagset data model, each measurement has a timestamp, and an associated set of tags (tagset) and set of fields (fieldset). The fieldset represents the actual measurement reading values, while the tagset represents the metadata to describe the measurements.Field data types are limited to floats, ints, strings, and booleans, and cannot be changed without rewriting the data. Tagset values are indexed, while fieldset values are not. Also, tagset values are always represented as strings, and cannot be updated.An example is below:An example tagset data model for time-series dataThe advantage of this approach is that if one’s data naturally fits the tagset model, then it is quite easy to get started, as one doesn’t have to worry about creating schemas or indexes.Conversely, the disadvantage of this model is that it is quite rigid and limited, with no ability to create additional indexes, indexes on continuous fields (e.g., numerics), update metadata after the fact, enforce data validation, etc.In particular, even though this model may feel “schemaless”, there is actually an underlying schema that is auto-created from the input data, which may differ from the desired schema.Data model summaryThe tagset data model in InfluxDB is more limiting and thus might be easier to get started with for some. However, the relational model in TimescaleDB is more versatile and offers more functionality, flexibility, and control. This is especially important as your application evolves. When planning your system you should consider both its current and future needs.Note: It’s also possible to create relational schema that are equivalent to the tagset model for specific use cases, such asPrometheusmetrics. For more on this, see theTimescale-Prometheus GitHub repository.Query languageGenerally in the world of database query languages, there have been two extremes: full SQL support on one end, and completely custom languages (sometimes known as “NoSQL”) on the other.SQL and Beyond: Timescale embraces SQL and extends it with special time-series functions. For more, see our detailed comparison onSQL vs Flux.From the beginning, TimescaleDB has firmly existed at the SQL end of the spectrum, fully embracing the language from day one, and later further extending it to simplify time-series analysis. This has enabled TimescaleDB to have a minimal learning curve for new users and allowed it to inherit the entire SQL ecosystem of third-party tools, connectors, and visualization options, which is larger than that of any other time-series database.In contrast, InfluxDB began with a “SQL-like” query language (calledInfluxQL), placing it in the middle of the spectrum, and then made a marked move towards the “custom” end with a new query language from InfluxDB calledFlux. (Read theFlux announcement on Hacker News, and our comparison ofSQL vs. Flux.)At a high level, here’s how the two language syntaxes compare, using a Flux query that performsmath across measurementsas an example:TimescaleDB (SQL)SELECT time, (memUsed / procTotal / 1000000) as value
FROM measurements
WHERE time > now() - '1 hour';SQL queryInfluxDB (Flux)// Memory used (in bytes)
memUsed = from(bucket: ""telegraf/autogen"")
  |> range(start: -1h)
  |> filter(fn: (r) =>
    r._measurement == ""mem"" and
    r._field == ""used""
  )
 
// Total processes running
procTotal = from(bucket: ""telegraf/autogen"")
  |> range(start: -1h)
  |> filter(fn: (r) =>
    r._measurement == ""processes"" and
    r._field == ""total""
    )
 
// Join memory used with total processes and calculate
// the average memory (in MB) used for running processes.
join(
    tables: {mem:memUsed, proc:procTotal},
    on: [""_time"", ""_stop"", ""_start"", ""host""]
  )
  |> map(fn: (r) => ({
    _time: r._time,
    _value: (r._value_mem / r._value_proc) / 1000000
  })
)Flux queryFor most use cases, we believe that SQL is the right query language for a time-series database. SQL has a rich tradition and history, including familiarity among millions of developers and a vibrant ecosystem of tutorials, training, and community leaders. In short, choosing SQL means you’re never alone.While Flux may make some tasks easier, there are significant trade-offs to adopting a custom query language. First, new query languages introduce significant overhead and reduce readability. They force a greater learning curve onto new users and possess a scarcity of compatible tools and community support.And they may not even be a viable option: rebuilding a system and re-educating a company to write and read a new query language is often not practically possible. Particularly if the company already is using SQL-compatible tools on top of the database, such as Tableau for visualization.This is also whySQL is making a comebackas the query language of choice for data infrastructure in general. Indeed, SQL is well-documented and is thethird-most commonly used programming language among developers.From the2020 Stack Overflow Developer SurveyReliabilityAnother cardinal rule for a database: it cannot lose or corrupt your data. This is a dimension where there is a stark difference in the approaches TimescaleDB and InfluxDB have taken, which has implications for reliability.At its start, InfluxDB sought to completelywrite an entire database in Go. In fact, it doubled down on this decision with its 0.9 release, which againcompletely rewrote the backend storage engine. WithInfluxDB 2.0(currently in beta at the time of publishing), it’s (at least) the second complete rewrite attempted by the InfluxData team.These design decisions have significant implications that affect reliability. First, InfluxDB has to implement the full suite of fault-tolerance mechanisms, including replication, high availability, and backup/restore. Second, InfluxDB is responsible for its on-disk reliability, e.g., to make sure all its data structures are both durable and resist data corruption across failures (and even failures during the recovery of failures).We made a dramatically different architectural decision when building TimescaleDB: build on PostgreSQL. TimescaleDB relies on the 25+ years of hard, careful engineering work that the entire PostgreSQL community has done to build a rock-solid database that supports millions of mission-critical applications worldwide.In fact, this was at the core of our co-founder’s launch post about TimescaleDB:When Boring is Awesome. Stateless microservices may crash and reboot, or trivially scale up and down. Actually, this is the entire “recovery-oriented computing” philosophy, as well as the thinking behind the new “serverless” design pattern. But your database needs to actually persist data, and should not wake you up at 3 a.m. because it’s in some broken state.So let us return to these two aspects of reliabilityFirst, programs can crash, servers can encounter hardware or power failures, disks can fail or experience corruption. You can mitigate this risk (e.g., robust software engineering practices, uninterrupted power supplies, disk RAID, etc.), but not eliminate it completely; it’s a fact of life for systems. In response, databases have been built with an array of mechanisms to further reduce such risk, including streaming replication to replicas, full-snapshot backup and recovery, streaming backups, robust data export tools, etc.Given TimescaleDB’s design, it’s able to leverage the full spectrum of tools that the Postgres ecosystem offers and has rigorously tested:streaming replicationfor high availability and read-only replicas,pg_dump and pg_recoveryfor full database snapshots,pg_basebackupand log shipping / streaming for incremental backups and arbitrary point-in-time recovery,pgBackrestorWAL-Efor continuous archiving to cloud storage, and robustCOPY FROMandCOPY TOtools for quickly importing/exporting data with a variety of formats.InfluxDB, on the other hand, has had to build all these tools from scratch. In fact, it doesn’t offer many of these capabilities even today. It initially offered replication and high availability in its open-source version but subsequently pulled this capability out of open source and into its enterprise product. InfluXDB backup tools have the ability to perform a full snapshot and recover to this point in time, and only recently added some support for a manual form of incremental backups.That said, InfluxDB’s approach of performing incremental backups based on database time ranges seems quite risky from a correctness perspective, given that timestamped data may arrive out-of-order, and thus the incremental backups -since some time period would not reflect this late data. And its ability to easily and safely export large volumes of data is also quite limited. We’ve heard from many users (including Timescale engineers in their past careers) that they had to write custom scripts to safely export data; asking for more than a few 10,000s of data points would cause the database to out-of-memory error and crash.The pain of trying to export data out of InfluxDB gave rise to Outflux, a tool to migrate data from InfluxDB to TimescaleDB with a single command.Second, databases need to provide strong on-disk reliability and durability, so that once a database has committed to storing a write, it is safely persisted to disk. In fact, for very large data volumes, the same argument even applies to indexing structures, which could otherwise take hours or days to recover; there’s good reason that file systems have moved from painful fsck recovery to journaling mechanisms.InTimescaleDB, we made the conscious decision not to change the lowest levels of PostgreSQL storage (even in implementinghybrid row/columnar storagein TimescaleDB native compression), nor interfere with the proper function of its write-ahead log (WAL). The WAL ensures that as soon as a write is accepted, it gets written to an on-disk log to ensure safety and durability, even before the data is written to its final location and all its indexes are safely updated. These data structures are critical for ensuring consistency and atomicity; they prevent data from becoming lost or corrupted, and ensure safe recovery. This is something the database community (and PostgreSQL) has worked hard on: what happens if your database crashes (and will subsequently try to recover) while it’s already in the middle of recovering from another crash?InfluxDB had to design and implement all recovery, reliability and durability functionality from scratch.This is a notoriously hard problem in databasesthat typically takes many years or even decades to get correct. Some metrics stores might be okay with occasionally losing data, but we see TimescaleDB being used in settings where this is not acceptable. InfluxDB forums, on the other hand, are rife with such complaints: “DB lost after restart,” “data loss during high ingest rate,” “data lost from InfluxDB databases,” “unresponsive due to corruption after disk disaster,” “data messed up after restoring multiple databases,” and so on.These challenges and problems are not unique to InfluxDB, and every developer of a reliable, stateful service must grapple with them. Every database goes through a period when it sometimes loses data because it's really, really hard to getallthe corner cases right. And eventually, all those corner cases come to haunt some operator. But, PostgreSQL went through this period in the 1990s, while InfluxDB is still figuring these things out today.PerformanceNow, let’s get into some hard numbers with a quantitative comparison of the two databases across a variety of insert and read workloads. Given how common high-cardinality datasets are within time series, we will first take a look at how InfluxDB and TimescaleDB handle this issue.Note: We've released all the code and data used for the below benchmarks as part of the open-source Time Series Benchmark Suite (TSBS) (GitHub,announcement).Machine configurationFor comparing both insert and read latency performance, we used the following setup:Version: TimescaleDBversion 1.7.1, community edition, with PostgreSQL 12, InfluxDBversion 1.8.0Open Source Edition (the latest non-beta releases for both databases at the time of publishing).One remote client machine, one database server, both in the same cloud data center.Instance size: both client and database server ran on DigitalOcean virtual machines (droplets) with 32 vCPU and 192 GB memory each.OS: both server and client machines ran Ubuntu 18.04.3Disk Size: 4 TB of block storage in a raid0 configuration (EXT4 filesystem), plus 800 GB of local SSD storage.Deployment method: database servers were deployed using Docker images, using images pulled from the official Docker hubs ofInflux DataandTimescale, respectively.Insert performanceFor insert performance, we used the following datasets and configuration:Dataset: 100-10,000,000 simulated devices generated 10 CPU metrics every 10 seconds for ~100M reading intervals. Intervals used for each configuration are as follows: 31 days for 100 devices, four days for 4,000 devices, three hours for 100,000 devices, and three minutes for 1,000,000 and 10,000,000 devices.Datasets used to benchmark insert performance between InfluxDB and TimescaleDBThe datasets were created usingTime-Series Benchmarking Suite, using thecpu-onlyuse case.Batch size: inserts were made using a batch size of 10,000 which was used for both InfluxDB and TimescaleDB.Additional database configurations: For TimescaleDB, we set the chunk time depending on the data volume, aiming for 7-16 chunks in total for each configuration (more on chunks here). For InfluxDB, we enabled theTSI (time series index). All other parameters were kept as default.Insert Rate Comparison: TimescaleDB outperforms InfluxDB as number of devices and cardinality of data increases.Performance comparison: Timescale is gives ~3.5x the performance of InfluxDB for high cardinality dataOn insert performance as the cardinality of the dataset increases, the results are fairly clear.For workloads with extremely low cardinality, like the configuration with 100 devices, InfluxDB offers better insert performance than TimescaleDB.However, as cardinality increases, InfluxDB performance drops dramatically due to its reliance on time-structured merge trees (which, similar to the log-structured merge trees it is modeled after, suffers with higher-cardinality datasets).This of course should be no surprise, as high cardinality is a well-known Achilles heel for InfluxDB(source:GitHub,Forums). In comparison, TimescaleDB actually sees better performance as cardinality increases, with moderate drop off in terms of absolute insert rate, very quickly surpassing InfluxDB in terms of insert performance for the configurations of 4,000, 100,000, 1 million, and 10 million devices.That said, it is worth doing an honest analysis of your insert needs. If your insert performance is far below these benchmarks (e.g., if it is 2,000 rows / second), then insert performance will not be your bottleneck, and this comparison becomes moot.Insert performance summaryFor workloads with extremely low cardinality, the databases are comparable, with InfluxDB outperforming Timescale.As cardinality increases, InfluxDB insert performance drops off dramatically faster than that with TimescaleDB.For workloads with high cardinality, TimescaleDB has ~3.5x the insert performance as InfluxDB.If your insert performance is far below these benchmarks (e.g., if it is 2,000 rows / second), then insert performance will not be your bottleneck.Read latencyFor benchmarking read latency, we used the following setup for each database(the machine configuration is the same as the one used in the insert comparison):Dataset: 100–4,000 simulated devices generated 1–10 CPU metrics every 10 seconds for four full days (100M+ reading intervals, 1B+ metrics).10,000 batch size was used for both on inserts.For TimescaleDB, we set the chunk time to 12 hours, resulting in eight total chunks (more on chunk time here).We also enablednative compressionon TimescaleDB, a new feature introduced in Timescale 1.5. We compressed all data older than 12 hours, resulting in seven chunks compressed and one chunk (data from the last 12 hours) uncompressed. This configuration is a commonly recommended one, since raw data is kept for recent time periods and older data is compressed, enabling greater query efficiency and ability to handle out-of-order data (see ourcompression docsfor more). The parameters we used to enable compression are as follows: we segmented by thetags_idandhostnamecolumns and ordered bytimedescending andusage_usercolumns.For InfluxDB, we enabled theTSI (Time Series Index).On read (i.e., query) latency, the results are more complex. Unlike inserts, which primarily vary on cardinality size (and perhaps batch size), the universe of possible queries is essentially infinite, especially with a language as powerful as SQL. Often, the best way to benchmark read latency is to do it with the actual queries you plan to execute. For this case, we use a broad set of queries to mimic the most common query patterns.The results shown below are the average from 1,000 queries for each query type. Latencies in this chart are all shown as milliseconds, with an additional column showing the relative performance of TimescaleDB compared to InfluxDB (highlighted in orange when TimescaleDB is faster, in blue when InfluxDB is faster).Results of benchmarking query performance between TimescaleDB and InfluxDBSIMPLE ROLLUPSFor simple rollups (i.e., groupbys), when aggregating one metric across a single host for 1 or 12 hours, or multiple metrics across one or multiple hosts (either for 1 hour or 12 hours), TimescaleDB generally outperforms InfluxDB at both low and high cardinality. In particular, TimescaleDB exhibited 460 % the performance of InfluxDB on configurations with 100 and 4,000 devices with 10 unique metrics being generated every read interval.AGGREGATESWhen calculating a simple aggregate for one device, performance is comparable between both TimescaleDB and InfluxDB across any number of devices. But TimescaleDB significantly outperforms InfluxDB when it's necessary to aggregate more than one metric. In our benchmark, TimescaleDB demonstrates 168 % the performance of InfluxDB when aggregating eight metrics across 100 devices, and 156 % when aggregating eight metrics across 4,000 devices. Once again, TimescaleDB outperforms InfluxDB for high-end scenarios.DOUBLE ROLLUPSFor double rollups aggregating metrics by time and another dimension (e.g., GROUPBY time, deviceId): When aggregating one metric, InfluxDB shows better performance than TimescaleDB with TimescaleDB only 54% as good as InfluxDB for the 4,000 device config. However, as the number of metrics being aggregated increases, TimescaleDB achieves 188% the performance of InfluxDB.THRESHOLDSWhen selecting rows based on a threshold, TimescaleDB outperforms InfluxDB. Timescale demonstrates between 350-860% the performance of InfluxDB when computing thresholds for a single device and 175-258% the performance of InfluxDB when computing thresholds for all devices for a random time window.COMPLEX QUERIESFor complex queries that go beyond rollups or thresholds, the comparison is much more clear-cut: TimescaleDBvastlyoutperforms InfluxDB here (in some cases, over thousands of times faster). Theabsolutedifference in performance here is actually quite stark: While InfluxDB might be faster by a few milliseconds or tens of milliseconds for some of the single-metric rollups, that difference is mostly indistinguishable to human-facing applications.For complex queries that go beyond rollups or thresholds, there really is no comparison: TimescaleDB vastly outperforms InfluxDB here (in some cases over thousands of times faster).Yet for these more complex queries, TimescaleDB provides real-time responses (e.g., 10–100s of milliseconds), while InfluxDB sees significant human-observable delays (tens of seconds).It’s worth noting that there were several other complex queries that we couldn’t test because of lack of support from InfluxDB: e.g., joins, window functions, geospatial queries, etc.Notice that Timescale exhibits 340-7100% the performance of Influx on these complex queries, many of which are common to historical analysis and monitoring.Read latency performance summaryFor simple queries, TimescaleDB generally outperforms InfluxDB.For aggregates and double roll-ups, TimescaleDB also generally outperforms InfluxDB. However, when simply rolling up just a single metric, InfluxDB can sometimes outperform TimescaleDB.When selecting rows based on a threshold, TimescaleDB outperforms InfluxDB by a significant margin, being up to 414 % faster.For complex queries, TimescaleDB vastly outperforms InfluxDB, and supports a broader range of query types; the difference here is often in the range of seconds to tens of seconds, with Timescale 344-7100 % the performance improvement over InfluxDB.Stability issues during benchmarkingWe had several operational issues benchmarking InfluxDB as our datasets grew, even with the Influx Time-series Index (TSI) enabled. In particular, as we experimented with higher cardinality data sets (100K+ tags), we ran into trouble with both inserts and queries on InfluxDB (but not on TimescaleDB).While we were able to insert batches of 10k into InfluxDB at lower cardinalities, once we got to 100k devices, we would experience timeouts and errors with batch sizes that large. The most common errors were write errors caused by exceeding the maximum cache memory size, timeouts, and fatal out-of-memory errors, which all occurred during runtime.An example of the fatal errors we encountered while benchmarking InfluxDBSolving these errors required a combination of increasing the maximum cache size, from the default 1 GB to between 4 GB and 64 GB as we went from 100,000 to 10,000,000 devices, as well as decreasing the batch size from 10,000 to between 5,000 and 1,000 and using client-side code to deal with the backpressure incurred at higher cardinalities. We had to force our client code to sleep for up to 30 seconds after requests received errors writing the batches.In contrast, with TimescaleDB, we were able to write large batches at higher cardinality without issue and with no additional configuration.Moreover, starting at 100K cardinality, we also experienced problems with some of our read queries on InfluxDB. Our InfluxDB HTTP connection would error out with a cryptic ‘End of File’ message. When we investigated the InfluxDB server, we found out that InfluxDB had consumed all available memory to run the query and subsequently crashed with an Out of Memory error. Since PostgreSQL helpfully allows us to limit system memory usage with settings likeshared_buffersandwork_mem, this generally was not an issue for TimescaleDB, even at higher cardinalities.High-Cardinality DatasetsHigh-cardinality datasets are a significant weakness for InfluxDB.InfluxDB and the TSIHigh-cardinality datasets are a significant weakness for InfluxDB. This is because of how the InfluxDB developers have architected their system, starting with their Time-series Index (TSI).The InfluxDB TSI is a home-grown log-structured merge tree-based system comprised of various data structures, including hashmaps and bitsets. This includes an in-memory log (“LogFile”) that gets periodically flushed to disk when it exceeds a threshold (5 MB) and compacted to an on-disk memory-mapped index (“IndexFile”); a file  (“SeriesFile”) that contains a set of all series keys across the entire database. (Describedhere in the InfluxDB documentation.)The performance of the TSI is limited by the complex interactions of all of these data structures.The design decisions behind the TSI also lead to several other limitations with performance implications:Their total cardinality limit is around 30 million (although based on the graph above, InfluxDB starts to perform poorly well before that), or far below what is often required in time-series use cases like IoT and IT Monitoring.InfluxDB indexes tags but not fields, which means that queries that filter on fields can not perform better than full scans. For example, if one wanted to search for all rows where there was no free memory (e.g, something like,SELECT * FROM sensor_data WHERE mem_free = 0), one could not do better than a full linear scan (i.e., O(n) time) to identify the relevant data points.The set of columns included in the index is completely fixed and immutable. Changing what columns in your data are indexed (tagged) and what things are not requires a full rewrite of your data.InfluxDB is only able to index discrete, and not continuous, values due to its reliance on hashmaps. For example, to search all rows where temperature was greater than 90 degrees (e.g., something likeSELECT * FROM sensor_data WHERE temperature > 90), one would again have to fully scan the entire dataset.Your cardinality on InfluxDB is affected by your cardinality across all time, even if some fields/values are no longer present in your dataset. This is because the SeriesFile stores all series keys across the entire dataset.TimescaleDB and B-treesIn contrast, TimescaleDB is a relational database that relies on a proven data structure for indexing data: the B-tree. This decision leads to its ability to scale to high cardinalities.First, TimescaleDB partitions your data by time, with one B-tree mapping time-segments to the appropriate partition (“chunk”). All of this partitioning happens behind the scenes and is hidden from the user, who is able to access a virtual table (“hypertable”) that spans all of their data across all partitions.Next, TimescaleDB allows for the creation of multiple indexes across your dataset (e.g., for equipment_id, sensor_id, firmware_version, site_id). These indexes are then created on every chunk, by default in the form of a B-tree. (One can also create indexes using any of the built-inPostgreSQL index types: Hash, GiST, SP-GiST, GIN, and BRIN.)This approach has a few benefits for high-cardinality datasets:The simpler approach leads to a clearer understanding of how the database performs. As long as the indexes and data for the dataset we want to query fit inside memory, which is something that can be tuned, cardinality becomes a non-issue.In addition, since the secondary indexes are scoped at the chunk level, the indexes themselves only get as large as the cardinality of the dataset for that range of time.You have control over which columns to index, including the ability to create compound indexes over multiple columns. You can also add or delete indexes anytime you want, for example, if your query workloads change. Unlike in InfluxDB, changing your indexing structure in TimescaleDB does not require you to rewrite the entire history of your data.You can create indexes on discrete and continuous fields, particularly because B-trees work well for a comparison using any of the following operators:<, <=, =, >=, >, BETWEEN, IN, IS NULL, IS NOT NULL. Our example queries from above (SELECT * FROM sensor_data WHERE mem_free = 0andSELECT * FROM sensor_data WHERE temperature > 90’)will run in logarithmic, or O(log n), time.The other supported index types can come in handy in other scenarios, e.g., GIST indexes for “nearest neighbor” searches.EcosystemThe database can only do so much, which is when one typically turns to the broader third-party ecosystem for additional capabilities. This is when the size and scope of the ecosystem make a large difference.TimescaleDB’s approach of embracing SQL pays large dividends, as it allows TimescaleDB to speak with any tool that speaks SQL. In contrast, the non-SQL strategy chosen by InfluxDB isolates the database and limits how InfluxDB can be used by its developers.Having a broad ecosystem makes deployment easier. For example, if one is already usingTableauto visualize data or Apache Spark for data processing, TimescaleDB can plug right into the existing infrastructure due to its compatible connectors.Here is a non-exhaustive list of first-party (e.g., the components of the InfluxData TICK stack) and third-party tools that connect with either database, to show the relative difference in the two database ecosystems.Official support refers to when tool makers themselves support the database–for example, the visualization tool Grafana has official support for both TimescaleDB and InfluxDB.Official support is given three checkmarks.Unofficial support refers to when toolmakers do not support the database natively in the tool, but a connector or library is available.For tools that give either database unofficial support, we differentiate the quality of those tools based on the number of GitHub stars they’ve received. Unofficial tools with less than 100 GitHub stars are given one checkmark, but those with 100 stars or more are given two checkmarks.For the open-source projects below, to reflect the popularity of the projects, we included the number of GitHub stars they had as of publication in parentheses, e.g., Apache Kafka (9k+). For many of the unofficial projects for InfluxDB, for example, the unofficial supporting project was often very early (very few stars) or inactive (no updates in months or years).Ecosystem Comparison: InfluxDB vs TimescaleDBOperational ManagementEven if a database satisfies all the above needs, it still needs to work, and someone needs to operate it.Based on our experience, operational management requirements typically boil down to these categories: high availability, resource consumption (memory, disk, CPU), general tooling.High availabilityNo matter how reliable the database, at some point, your node will go down: hardware errors, disk errors, or some other unrecoverable issue.Thus, high availability of your database goes from a value-added feature to a requirement for production deployments. At that point, you will want to ensure you have a stand-by available for failover with no loss of data.While InfluxDB's high availability is only offered by their paidenterprise version, TimescaleDB supports high availability for free in both its open-source and Community editions, via PostgreSQL streaming replication (as explained inthis tutorial). This is yet another benefit that Timescale inherits as a result of the rock-solid foundation of PostgreSQL.Resource consumptionDisk usageInfluxDB offers native compression using avariety of techniques: Snappy for strings, delta encoding, scaling, and simple-8b with run length encoding for timestamps, Gorilla delta encoding for floats, bits for booleans, and delta encoding for integers.TimescaleDB offersnative compressionusing a novel hybrid row/columnar storage approach, using Gorilla compression for floats, delta-of-delta, and simple-8b with run-length encoding for timestamps and integer-like types, whole-row dictionary compression for a few repeating values, with LZ compression on top and lastly LZ-based array compression for all other types. (See here for more onhow TimescaleDB’s native compression works.)Using a combination of datasets for the insert and query benchmarks above, and compressing all chunks for TimescaleDB, here’s how the two databases fared at varying cardinalities:Disk Usage Comparison: InfluxDB vs TimescaleDBNote: Numbers do not include WAL size, as that is configurable by the user.Thanks to its column-oriented structure, InfluxDB is able to achieve better compression ratios overall. However, the gap is on the order of megabytes on the low end and narrows at higher cardinalities.Considering the low memory usage of TimescaleDB compared to InfluxDB, and the fact that memory is often an order of magnitude more expensive than disk, we have found that this gap in storage not to be an issue.CPU usageAccording to anexternal comparisonbyDNSFilter, using TimescaleDB resulted in 10x better resource utilization (even with 30 % higher requests) when compared to InfluxDB:Source:DNSFilter Comparison (March 2018)General toolingWhen operating TimescaleDB, one inherits all of the battle-tested tools that exist in the PostgreSQL ecosystem:pg_dumpandpg_restorefor backup/restore, HA/failover tools likePatroni, load balancing tools for clustering reads likePgpool, etc. Since TimescaleDB looks and feels like PostgreSQL, there are minimal operational learning curves. TimescaleDB “just works,” as one would expect from PostgreSQL.For operating InfluxDB, one is limited to thetools that the Influx team has built: backup, restore, internal monitoring, etc.Company and Community SupportFinally, when investing in an open-source technology primarily developed by a company, you are implicitly also investing in that company’s ability to serve you, whether you’re a paying customer or not. With that in mind, let’s note the differences between Timescale and InfluxData, the companies behind TimescaleDB and InfluxDB.Timescale continues to invest in the community with its free self-managed versions while also actively developing itshigh-performance cloud offering for time series and analytics. Timescale announced that multi-node scale-out will be available for free. (Seethis postto learn more about TimescaleDB multi-node design and availability, as well as more ways Timescale is investing in its community.)In contrast, while Influx does have an open-source offering alongside its licensed InfluxDB Enterprise, its more advanced features, likeclustering, remain gated behind an enterprise license. The company also appears to bede-prioritizing its open-source productand instead focusing on their SaaS offering, Influx Cloud.Moreover, for technical products, support and resources often come not just from the company building the technology but the community of developers who use it. InfluxData is building their community from scratch, while Timescale is able to inherit and build on PostgreSQL’s community. This means that InfluxData community support is a walled garden: inherently more closed, less varied, and smaller compared to the open diversity of expertise present in the vibrant 20+ year-old PostgreSQL ecosystem.Furthermore, because TimescaleDB operates just like PostgreSQL, much of the knowledge base relevant to PostgreSQL is also relevant to TimescaleDB. So if you are new to TimescaleDB (or SQL or PostgreSQL), there are many resources available to help get you started. Alternatively, if you are already a SQL or PostgreSQL expert, you will already know how to use the majority of TimescaleDB (save for a small learning curve of optimizations built specifically for time-series data, like SQL functions for complex analysis).SummaryNo one wants to invest in technology only to have it limit their growth or scale in the future, let alone invest in something that's the wrong fit today.Before making a decision, take a step back and analyze your stack, your team's skills, and your needs (now and in the future). It could be the difference between infrastructure that evolves and grows with you and one that crumbles to the ground and forces you to start all over.In this post, we performed a detailed comparison of TimescaleDB and InfluxDB. We don’t claim to be InfluxDB experts, so we’re open to any suggestions on how to improve this comparison. In general, we aim to be as transparent as possible about our data models, methodologies, and analysis, and we welcome feedback. We also encourage readers to raise any concerns about the information we’ve presented in order to help us with benchmarking in the future.We recognize thatTimescaleDBisn’t the only time-series solution, and there are situations where it might not be the best answer. And we strive to be upfront in admitting where an alternate solution may be preferable. But we’re always interested in holistically evaluating our solution against that of others, and we’ll continue to share our insights with the greater community.Like this post? Please recommend and/or share!Keep LearningIf you want to keep reading about TimescaleDB, these articles will help you learn more:An introduction to hypertablesMore time-series data, less lines of code: meet hyperfunctionsUsing materialized views for time-series data analysisAllowing DML operations in highly-compressed data via Timescale compressionTimescaleDB performance benchmark vs Amazon RDS PostgreSQL📝 Are you migrating off InfluxDB? We can help.Check outOutflux, our migration tool, andreach out to us for expert migration advice.Our Support team has helped many others migrate from InfluxDB to Timescale successfully.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescaledb-vs-influxdb-for-time-series-data-timescale-influx-sql-nosql-36489299877/
2019-07-31T17:00:07.000Z,OrderedAppend: An optimization for range partitioning,"With this feature, we’ve seen up to 100x performance improvements for certain queries.In our previous post onimplementing constraint exclusion, we discussed how TimescaleDB leverages PostgreSQL’s foundation and expands on its capabilities to improve performance. Continuing with the same theme, in this post we will discuss how we’ve added support for ordered appends which optimize a large range of queries, particularly those that are ordered by time.We’ve seen performance improvements up to 100x for certain queries after applying this feature, so we encourage you to keep reading!Optimizing Appends for large queriesPostgreSQL represents how plans should be executed using “nodes”. There are a variety of different nodes that may appear in an EXPLAIN output, but we want to focus specifically on Append nodes, which essentially combine the results from multiple sources into a single result.PostgreSQL has two standard Appends that are commonly used that you can find in an EXPLAIN output:Append:appends results of child nodes to return a unioned resultMergeAppend:merge output of child nodes by sort key; all child nodes must be sorted by that same sort key; accesses every chunk when used in TimescaleDBWhen MergeAppend nodes are used with TimescaleDB, we necessarily access every chunk to figure out if the chunk has keys that we need to merge. However, this is obviously less efficient since it requires us to touch every chunk.To address this issue, with the release ofTimescaleDB 1.2we introducedOrderedAppendasan optimization for range partitioning. The purpose of this feature is to optimize a large range of queries, particularly those that are ordered by time and contain a LIMIT clause. This optimization takes advantage of the fact that we know the range of time held in each chunk, and can stop accessing chunks once we’ve found enough rows to satisfy the LIMIT clause. As mentioned above, with this optimization we see performance improvements of up to 100x depending on the query.With the release ofTimescaleDB 1.4, we wanted to extend the cases in which OrderedAppend can be used. This meant making OrderedAppend space-partition aware, as well as removing the LIMIT clause restriction from Ordered Append. With these additions, more users can benefit from the performance benefits achieved through leveraging OrderedAppend.(Additionally, the updates to OrderedAppend for space partitions will be leveraged even more heavily with the release ofTimescaleDB clusteringwhich is currently in private beta. Stay tuned for more information!)Developing query plans with the optimizationAs an optimization for range partitioning, OrderedAppend eliminates sort steps because it is aware of the way data is partitioned.Since each chunk has a known time range it covers to get sorted output, no global sort step is needed. Only local sort steps have to be completed and then appended in the correct order. If index scans are utilized, which return the output sorted, sorting can be completely avoided.For a query ordering by the time dimension with a LIMIT clause you would normally get something like this:dev=# EXPLAIN (ANALYZE,COSTS OFF,BUFFERS,TIMING OFF,SUMMARY OFF)
dev-# SELECT * FROM metrics ORDER BY time LIMIT 1;
                                                 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Limit (actual rows=1 loops=1)
   Buffers: shared hit=16
   ->  Merge Append (actual rows=1 loops=1)
         Sort Key: metrics.""time""
         Buffers: shared hit=16
         ->  Index Scan using metrics_time_idx on metrics (actual rows=0 loops=1)
               Buffers: shared hit=1
         ->  Index Scan using _hyper_1_1_chunk_metrics_time_idx on _hyper_1_1_chunk (actual rows=1 loops=1)
               Buffers: shared hit=3
         ->  Index Scan using _hyper_1_2_chunk_metrics_time_idx on _hyper_1_2_chunk (actual rows=1 loops=1)
               Buffers: shared hit=3
         ->  Index Scan using _hyper_1_3_chunk_metrics_time_idx on _hyper_1_3_chunk (actual rows=1 loops=1)
               Buffers: shared hit=3
         ->  Index Scan using _hyper_1_4_chunk_metrics_time_idx on _hyper_1_4_chunk (actual rows=1 loops=1)
               Buffers: shared hit=3
         ->  Index Scan using _hyper_1_5_chunk_metrics_time_idx on _hyper_1_5_chunk (actual rows=1 loops=1)
               Buffers: shared hit=3You can see 3 pages are read from every chunk and an additional page from the parent table which contains no actual rows.While with this optimization enabled you would get a plan looking like this:dev=# EXPLAIN (ANALYZE,COSTS OFF,BUFFERS,TIMING OFF,SUMMARY OFF)
dev-# SELECT * FROM metrics ORDER BY time LIMIT 1;
                                                 QUERY PLAN
------------------------------------------------------------------------------------------------------------
 Limit (actual rows=1 loops=1)
   Buffers: shared hit=3
   ->  Custom Scan (ChunkAppend) on metrics (actual rows=1 loops=1)
         Order: metrics.""time""
         Buffers: shared hit=3
         ->  Index Scan using _hyper_1_1_chunk_metrics_time_idx on _hyper_1_1_chunk (actual rows=1 loops=1)
               Buffers: shared hit=3
         ->  Index Scan using _hyper_1_2_chunk_metrics_time_idx on _hyper_1_2_chunk (never executed)
         ->  Index Scan using _hyper_1_3_chunk_metrics_time_idx on _hyper_1_3_chunk (never executed)
         ->  Index Scan using _hyper_1_4_chunk_metrics_time_idx on _hyper_1_4_chunk (never executed)
         ->  Index Scan using _hyper_1_5_chunk_metrics_time_idx on _hyper_1_5_chunk (never executed)After the first chunk, the remaining chunks never get executed and to complete the query only 3 pages have to be read. TimescaleDB removes parent tables from plans like this because we know the parent table does not contain any data.MergeAppend vs. ChunkAppendThe main difference between these two examples is the type of Append node we used. In the first case, a MergeAppend node is used. In the second case, we used a ChunkAppend node (also introduced in 1.4) which is a TimescaleDB custom node that works similarly to the PostgreSQL Append node, but contains additional optimizations.The MergeAppend node implements the global sort and requires locally sorted input which has to be sorted by the same sort key. To produce one tuple, the MergeAppend node has to read one tuple from every chunk to decide which one to return to.For the very simple example query above, you will see 16 pages read (with MergeAppend) vs. 3 pages (with ChunkAppend) which is a 5x improvement over the unoptimized case (if we ignore the single page from the parent table), and represents the number of chunks present in that hypertable. So for a hypertable with 100 chunks, there would be 100 times less pages to be read to produce the result for the query.As you can see, you gain the most benefit from OrderedAppend with a LIMIT clause as older chunks don’t have to be touched if the required results can be satisfied from more recent chunks. This type of query is very common in time-series workloads (e.g. if you want to get the last reading from a sensor). However, even for queries without a LIMIT clause, this feature is beneficial because it eliminates sorting of data.Next stepsIf you are interested in using OrderedAppend, make sure you have TimescaleDB 1.2 or higher installed (installation guide). However, we always recommend upgrading to the most recent version of the software (at the time of publishing this post, it’sTimescaleDB 1.4).If you are brand new to TimescaleDB, get startedhere. Have questions? Join ourSlackchannel or leave them in the comments section below.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/ordered-append-an-optimization-for-range-partitioning/
2019-08-08T15:20:32.000Z,Simplified Time-Series Analytics Using the time_bucket() Function,"What Is Time-Series Analytics?Time-series analytics refers to the process of analyzing and extracting insights from data that is organized and collected over time, typically at regular intervals—or, in other words, time-series data.If you are working with time-series data, you need a way to be able to easily manipulate, query, and visualize that data to perform time-series analytics. What you may or may not already know is thatTimescaleDBprovides a number of time-oriented functions thataren't found in traditional relational databases.These functions are meant to provide two key benefits: improved ease of use for time-series analytics and improved performance. The functions live alongside full SQL and can be viewed as extensions of the SQL language. Since many developers today already know SQL, the learning curve is greatly reduced.Two critical TimescaleDB time-series functions are:time_bucket()andtime_bucket_gapfill(). Time_bucket() is used for aggregating arbitrarily-sized time periods, and gapfill() is important when your time buckets have missing data or gaps, which is a very common occurrence when capturing thousands of time-series readings per second. Together, both of these are essential for analyzing and visualizing time-series data.In this blog, we’ll discuss both of these capabilities and show them in action using Grafana.Time-series analytics: Background ontime_bucket()Essentially,time_bucket()is a more powerful version of the standard PostgreSQLdate_trunc()function.date_trunc“truncates” a TIMESTAMP or an INTERVAL value based on a specified date part (e.g., hour, week, or month) and returns the truncated timestamp or interval.For example,date_trunccan aggregate by one second, one hour, one day, or one week. However, users often want to see aggregates by five minutes or four hours, etc. This can get pretty complicated in SQL, buttime_bucket()makes it easy.Time bucketing allows for arbitrary time intervals (e.g., five minutes, six hours, etc.), as well as flexible groupings and offsets, instead of just second, minute, hour, and so on.In addition to allowing more flexible time-series queries,time_bucket()also allows you to write these queries in a simpler way. In fact, it can infer the time range directly from the WHERE clause, which greatly simplifies the query syntax.Here it is in action:SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
  FROM metrics
  GROUP BY five_min
  ORDER BY five_min DESC LIMIT 12;By the way, this time-series analytics feature has actually been a core function of TimescaleDB since its first release in April 2017, and our users love it, particularly for time-series analytics!When to Usetime_bucket()for Time-Series AnalyticsAs you can imagine, time bucketing can be helpful for a number of scenarios. When it comes to creating dashboards or visualizations of time-series data, many rely on this function to turn their raw observations into fixed time intervals.When graphing time-series data using a solution such asGrafana, aggregations can help identify trends over time by grouping raw data into higher-level aggregates. For example, you might want to average monthly raw data daily to achieve a smoother trend line or count the number of occurrences of non-numeric data.Building on top of time_bucket() with time_bucket_gapfill()One issue users often encounter when working with time-series data is recording measurements at irregular or mismatched intervals. This creates a time bucket interval with missing data or gaps.Fortunately, TimescaleDB has a function calledtime_bucket_gapfill()that allows you to aggregate your data into continuous time intervals. You can choose two different ways to fill in the gaps:locf()which carries the last known value in the time range forward orinterpolate()which does a linear interpolation between gaps.Time_bucket_gapfillwas introduced in TimescaleDB 1.2 as a Community feature. To learn more about gapfill, check out thisblog post.Get Hands-On With a Sample ApplicationNow that you know whattime_bucket()andtime_bucket_gapfill()do, it’s time to get hands-on with a sample application for time-series analytics!We’ve built a simple Python application that pulls data from the Open AQ Platform API (an open API air quality data). Essentially, the application reads the API to get air quality values for all cities in Great Britain over a period of time, parses the results, and stores all measurements collected from air quality sensors in TimescaleDB.All the code for this application can be found on GitHubhttps://github.com/timescale/examples/tree/master/air-quality.We also connected a Grafana instance to the TimescaleDB database that stores these results and wrote a step-by-step tutorial on how to usetime_bucket()andtime_bucket_gapfill()to visualize the data.You can build your own dashboard usingTimescale(which runs both TimescaleDB and Grafana)!Next StepsThetime_bucket()function is just one example of how TimescaleDB offers optimized SQL functions for working with time-series data and time-series analytics. To learn more about other TimescaleDB functions, check out ourdocs.If you’re ready to get started, you candownload TimescaleDBor sign up forTimescaleto quickly get Timescale and Grafana instances up and running (free 30-day trial, no credit card required).Like this post and are interested in learning more? Sign up for our mailing list below (right-hand side of the screen), or follow us onTwitter.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/simplified-time-series-analytics-using-the-time_bucket-function/
2020-09-15T17:14:54.000Z,Top 5 PostgreSQL Extensions,"PostgreSQL is designed to be easily extensible, andPostgreSQL extensionsadd extra functionality to your database by modifying and enhancing how it does certain processes.  Moreover, Postgres extensions can help with some of the limitations you may find with vanilla Postgres (such as working efficiently with time-series data)—withoutthe hassle of switching to a whole new database.✨Explore some of these andother PostgreSQL extensionsin our guide.Extensions loaded into the database can function just like built-in features, anda rich ecosystem of extensionsfor all manner of use cases has sprung up over the years. For example, extensions can enable you to handle time series and geospatial data, track the performance of your queries, and more.It can be tricky to know which extensions are the best or most relevant ones to use, so we did the research for you.To compile our list, we asked our team andTimescaleDB Slackcommunity members to recommend their favorite PostgreSQL extensions. The below includes most of the frequently recommended extensions, as well as some less widely known— but useful—ones too.We’ve also included sample queries to help you see how each extension works, what it allows you to do with your data, and determine which ones might be right for you.1. TimescaleDBWe're biased, but TimescaleDB tops our list (developers agree with us: there are 500K+ active TimescaleDB deployments around the world 🤗).What is it, and why use it?TimescaleDB is “Postgres for time-series.” It’s packaged as a Postgres extension and is purpose-built for time-series use cases. The most common uses for TimescaleDB involve storing massive amounts of data for cloud infrastructure metrics, product analytics, web analytics, and IoT devices, but there are many more.These use cases are time-centric, almost solely append-only (lots of INSERTs), and require rapid ingestion of large amounts of data at high ingestion rates during small time windows.Check out ourWhat Is Time-Series Data? Definitions & Examplesblog post to learn more.TimescaleDB also supports full SQL, enables fast, complex queries on your data, and has a variety of other features for simplifying monitoring and analysis, such asreal-time aggregationandautomated data retention policies. With TimescaleDB, you can also join your time-series data with your relational business data to gain additional insight, something that’s hard to do with pure metrics databases like CrateDB or InfluxDB.InstallationThe best way to get TimescaleDB is viaTimescale, our fully managed database-as-a-service product. You can also download TimescaleDB and run it on your own infrastructure.👉Create a Timescale accountorinstall TimescaleDB.TimescaleDB sample queryWe’ll use a sample query that analyzes taxi rides during the month of January 2016.This query has both time-series and relational components and is a good example of how TimescaleDB allows you to combine time-series data with relational metadata for more insightful analysis.We want to know how many rides occur over a day (time-series data) for each rate type (metadata about the rate the customer paid).SELECT time_bucket('5 minute', pickup_datetime) AS five_min, count(*), rates.description
FROM rides
JOIN rates ON rides.rate_code = rates.rate_code
WHERE rides.rate_code IN (2,3) AND pickup_datetime < '2016-01-02'
GROUP BY five_min, rates.description 
ORDER BY five_min
LIMIT 4;We get the following results, showing the number of rides that took place in each 5-minute time bucket for each rate type:-[ RECORD 1 ]--------------------
five_min    | 2016-01-01 00:00:00
count       | 1
description | Newark
-[ RECORD 2 ]--------------------
five_min    | 2016-01-01 00:00:00
count       | 3
description | JFK
-[ RECORD 3 ]--------------------
five_min    | 2016-01-01 00:05:00
count       | 1
description | Newark
-[ RECORD 4 ]--------------------
five_min    | 2016-01-01 00:05:00
count       | 6
description | JFK2. PostGISWhat is it, and why use it?PostGIS extends Postgres to handle spatial data and data types. It adds support for geographic objects, allowing location queries to be run in SQL. PostGIS adds extra types (like geometry, geography, raster, and others) to the Postgre database, as well as functions, operators, and index enhancements that apply to these spatial types. These additional functions, operators, index bindings, and types augment the power of Postgres, making it a fast, feature-plenty, and robust spatial database management system.A core benefit of PostGIS is that geospatial queries can take place at the database level rather than the application level, making them more efficient. PostGIS is useful for many IoT use cases that involve tracking, routing, addressing, and other location-based attributes.Installation👉Get PostGISPostGIS sample queryFor this one, we'll use a sample query that asks a question about geospatial and time-series elements. It's a great example of how you might combine different Postgres extensions to meet your requirements (in this case, combining PostGIS and TimescaleDB).We want to know: ""How many taxis picked up passengers within 400m of Times Square on New Year's Day 2016?""-- How many taxis pick up rides within 400m of Times Square on New Years Day, grouped by 30 minute buckets.
-- Note: Times Square is at (lat, long) (40.7589,-73.9851)

SELECT time_bucket('30 minutes', pickup_datetime) AS thirty_min, COUNT(*) AS near_times_sq
FROM rides
WHERE ST_Distance(pickup_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-73.9851,40.7589),4326),2163)) < 400
AND pickup_datetime < '2016-01-01 14:00'
GROUP BY thirty_min ORDER BY thirty_min
LIMIT 5;The above query produces the following results:-[ RECORD 1 ]-+--------------------
thirty_min    | 2016-01-01 00:00:00
near_times_sq | 74
-[ RECORD 2 ]-+--------------------
thirty_min    | 2016-01-01 00:30:00
near_times_sq | 102
-[ RECORD 3 ]-+--------------------
thirty_min    | 2016-01-01 01:00:00
near_times_sq | 120
-[ RECORD 4 ]-+--------------------
thirty_min    | 2016-01-01 01:30:00
near_times_sq | 98
-[ RECORD 5 ]-+--------------------
thirty_min    | 2016-01-01 02:00:00
near_times_sq | 112👉 See how one company,Blue Sky Analytics, combines geospatial and time-series data to power their environmental monitoring platform.3. pg_stat_statementsWhat is it, and why use it?pg_stat_statements tracks statistics on the queries executed by a Postgres database. It will help you debug queries, identify slow queries, and generally give you deeper information about how your queries are running. The statistics gathered by the module are made available via a system view named pg_stat_statements.Install👉 pg_stat_statements ships with most versions of Postgres.To enable the extension, run the below on your target database, using psql or your database administration tool of choice:CREATE EXTENSION pg_stat_statements;pg_stat_statements sample querypg_stat_statements allows us to uncover lots of useful information, but we’ll share just one example.In the query below, we calculate the total and average execution time for all of our queries and show the 100 queries that take the longest:SELECT (total_time / 1000 / 60) as total, 
(total_time/calls) as avg, 
query
FROM pg_stat_statements
ORDER BY 1 DESC
LIMIT 100;👉Learn more about pg_stat_statements4. ZomboDBWhat is it and why use it?ZomboDB enhances your ability to search text data in Postgres, making indexing and maintaining search data simpler and easier. It uses Elasticsearch as an index type to bring powerful text-search and analytics features to Postgres. Its comprehensive query language and SQL functions enable new and creative ways to query your relational data.ZomboDB abstracts away Elasticsearch such that it appears as a regular Postgres index. As a native Postgres index type, ZomboDB allows you toCREATE INDEX ... USING ZomboDBon your existing Postgres tables. At that point, ZomboDB takes over and fully manages the remote Elasticsearch index and guarantees transactionally-correct text-search query results.ZomboDB is worth exploring if Postgres'tsearchfeatures aren’t adequate for your requirements for searching large text content.Install👉Get ZomboDBZomboDB sample queryLet’s use the scenario of eCommerce to delve into the power of ZomboDB. Take this sample table calledproducts, representing a catalog of products:postgres=# SELECT * from products;

-[ RECORD 1 ]-----+-----------------------------------------------------------------------------------------------------------
id                | 1
name              | Magical Widget
keywords          | {magical,widget,round}
short_summary     | A widget that is quite magical
long_description  | Magical Widgets come from the land of Magicville and are capable of things you can't imagine
price             | 9900
inventory_count   | 42
discontinued      | f
availability_date | 2015-08-31
-[ RECORD 2 ]-----+-----------------------------------------------------------------------------------------------------------
id                | 2
name              | Baseball
keywords          | {baseball,sports,round}
short_summary     | It's a baseball
long_description  | Throw it at a person with a big wooden stick and hope they don't hit it
price             | 1249
inventory_count   | 2
discontinued      | f
availability_date | 2015-08-21
-[ RECORD 3 ]-----+-----------------------------------------------------------------------------------------------------------
id                | 3
name              | Telephone
keywords          | {communication,primitive,""alexander graham bell""}
short_summary     | A device to enable long-distance communications
long_description  | Use this to call your friends and family and be annoyed by telemarketers.  Long-distance charges may apply
price             | 1899
inventory_count   | 200
discontinued      | f
availability_date | 2015-08-11
-[ RECORD 4 ]-----+-----------------------------------------------------------------------------------------------------------
id                | 4
name              | Box
keywords          | {wooden,box,""negative space"",square}
short_summary     | Just an empty box made of wood
long_description  | A wooden container that will eventually rot away.  Put stuff it in (but not a cat).
price             | 17000
inventory_count   | 0
discontinued      | t
availability_date | 2015-07-01ZomboDB allows you to run queries using Elasticsearch syntax, like the one below, where we ask “Show me all products with the words ‘sports’ or ‘box’ in any fields”SELECT * FROM products WHERE products ==> 'sports box';On our table above, this query would return the following:-[ RECORD 1 ]-----+------------------------------------------------------------------------------------
id                | 2
name              | Baseball
keywords          | {baseball,sports,round}
short_summary     | It's a baseball
long_description  | Throw it at a person with a big wooden stick and hope they don't hit it
price             | 1249
inventory_count   | 2
discontinued      | f
availability_date | 2015-08-21
-[ RECORD 2 ]-----+------------------------------------------------------------------------------------
id                | 4
name              | Box
keywords          | {wooden,box,""negative space"",square}
short_summary     | Just an empty box made of wood
long_description  | A wooden container that will eventually rot away.  Put stuff it in (but not a cat).
price             | 17000
inventory_count   | 0
discontinued      | t
availability_date | 2015-07-01👉 To learn more about ZomboDB, see theirgetting started tutorial.5. Postgres_fdwWhat is it, and why use it?The postgres_fdw module enables you to use aForeign Data Wrapperto access tables on remote Postgres servers (hence the name ""fdw""). A Foreign Data Wrapper lets you create proxies for data stored in other Postgres databases, so that they may be queried as if they were coming from a table in the current database.Postgres_fdw allows you to combine data between two Postgres instances.Here’s a sample use case:You have a Postgres instance (A), and you use postgres_fdw to access data on Postgres instance (B), which is a remote instance.You then run queries that combine data from instances A and B at the database level rather than at the application level.Install👉  Get postgres_fdw by running the below from your psql command line:CREATE EXTENSION postgres_fdw IF NOT EXISTS;postgres_fdw sample queryCreate a connection to your foreign serverCREATE SERVER myserver FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '123.45.67.8', dbname ‘postgres’, port '5432');Creates a connection to a database hosted on ip address 123.45.67.8, with the name postgres at port 5432.Create user mapping so that users on your database can access the foreign server:CREATE USER MAPPING FOR postgres 
SERVER myserver 
OPTIONS (user 'postgres', password 'password');Once that’s done, you can import a schema from your foreign database and access any table.CREATE SCHEMA schema1;

IMPORT FOREIGN SCHEMA public
FROM SERVER myserver
INTO schema1;Once imported, you can now access tables of the foreign database on your ‘local’ database, like the example below where we access themetricstable:SELECT * FROM schema1.metrics
WHERE time < now() - ‘2 days’ :: interval;👉Follow this tutorialto learn more about postgres_fdw.⭐ BonusOther community suggestions that deserve honorable mentions:PLV8, a shared library that provides a Postgres procedural language powered by V8 Javascript Engine.Pg_repack, which helps you maintain tables at peak performance and perform online reindexing on your database.Learn MoreLooking to learn more about extending Postgres for scale and times-series scenarios?Check out the tutorials in Timescale's documentationto get started.Looking to join a community of TimescaleDB users and time-series enthusiasts?Join the TimescaleDB slack communityto meet like-minded developers, ask questions, and get help from the Timescale team (you’ll find me, our co-founders, and engineers active on all channels).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/top-5-postgresql-extensions/
2019-12-12T19:21:29.000Z,New: Helm Charts for Deploying TimescaleDB on Kubernetes,"Quickly & easily add TimescaleDB to your Kubernetes deployment strategy✨ Update: As of February 4, 2021, TimescaleDB 2.0 and multi-node is officially Generally Available ✨.Seeour announcement postfor details.The world of software moves fast, and we have seen some remarkable evolutionary cycles around how to deploy applications: with a move away from the monolithic stacks (bare metal, VM, container), down to the deployment and resilience of the applications core components (microservices).As developers, we know that it’s critical to bring the best software and customer experience to market as fast and reliably as possible. At Timescale, we’re committed to building the best time-series database for all developers and scenarios – which is why our team built the ability to deploy TimescaleDB alongside other cloud-native technologies.As a first step, we’ve released Helm Charts to help with deploying TimescaleDB on Kubernetes, which we will cover in this post.Download them today from GitHub(released under the Apache 2 open-source license).Enter KubernetesAs orchestration technology allows us to embrace cloud-native architectures, Kubernetes is a (disputablythe) core piece of managing and orchestrating microservice architectures. Early in my tenure with Timescale, we decided to invest in TimescaleDB’s native support for such technology (hat tip to Timescale engineersFeike Steenbergen,Ian Davis, and the rest of our cloud team for the heavy lifting).To start to turn our strategy into reality,  the TimescaleDB Cloud team developed a set ofHelm Charts, which allows you to quickly and easily add TimescaleDB to your Kubernetes deployment alongside other cloud native-technologies.Let's take a closer look at what deployment using TimescaleDB Helm Charts looks like:In the diagram above, we are deploying the service with three pods, a primary, and two replicas. Our Helm charts also deploy the Patroni agent – originally co-developed by Feike – to facilitate automatic leader election and failover amongst replicas for high availability. If the primary ever goes unavailable, a replica will automatically take over as the new primary and the cluster will automatically reconfigure itself to avoid any downtime.This (the diagram above) represents our single-primary deployment option which generates built-in availability via nodes for failover and the ability to horizontally scale the read operations.Note: to ease any storage-based I/O contention, we are deploying both a data volume and WAL volume for each node. Disk volumes are also remote from the actual database pod as well (e.g., as EBS volumes on AWS), so that even if the pod fails, k8s can automatically bring up a new instance of the database, reconnect it to the detached disk volumes, recover the database, and reintegrate it into the TimescaleDB cluster. All automatically.Now let's take a look at what TimescaleDB will look like if we use a hosted Kubernetes service, like Amazon Elastic Kubernetes Service (EKS):Again, you see our three-node deployment with a primary and two replicas deployed. However,  we’ve built a few things into the deployment that are worth mentioning.When deploying on AWS EKS:The pods will be scheduled on nodes that run in different Availability Zones (AZs).An AWS Elastic Load Balancer (ELB) is configured to handle routing incoming traffic to the Master pod.When configured for Backups to S3:Each pod will also include a container runningpgBackRest.By default, twoCronJobsare created to handle full weekly and incremental daily backups.The backups are stored to an S3 bucket.As you can see, we have built this out so that you are able to deploy to a cloud-native service and leverage all that it has to offer (i.e availability zones, backup, storage, etc.).What’s next?As we mentioned above, building the Helm Charts was just the first step, and you can check them out todayvia our public GitHub. This project is currently in active development.We’d love the Timescale Community’s feedback on the progress we’ve made so far. To do this, please either:File anissuein GitHub (within the TimescaleDB Kubernetes repository)Reach out to us via our communitySlackchannelWe have a similar set of Helm Charts available for deployment with ourmulti-node offering(available in the sameGitHub project; read theblog postfor more details). This allows you to horizontally scale your TimescaleDB instance, while deploying it as a cloud-native application.We’re excited to offer these deployment options to our users and are keen to hear about your experiences (and suggestions!) as you travel down the microservices path with Timescale.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/new-helm-charts-for-deploying-timescaledb-on-kubernetes/
2020-01-23T19:39:14.000Z,How to Proactively Manage Long-Term Data Storage With Downsampling,"Eliminate the need to store billions of rows with TimescaleDB’s upgraded continuous aggregatesIf you live in the world of monitoring and observability, you know that it requires the collection of time-series data at a massive scale and in as close to real-time as possible to understand system and application health and performance. As a result, collecting data in near real-time granularity generates very large data sets that can be difficult and costly to manage, and create significant database performance problems.Many monitoring tools address these issues by trading long term analytical value for reduced storage cost and improved performance. With aggressive default data retention policies and simply purging “historical” data from the system, these monitoring tools reduce data storage cost and protect performance, but they also eliminate the ability to extract long term analytical value from the data.With TimescaleDB, you don't have to sacrifice long term analytics of your monitoring data to reduce storage costs and improve performance. Instead, TimescaleDB allows you to reduce the granularity of your data by aggregating data into coarser periods of time and dropping the finer-grained real-time data while maintaining data accuracy. This allows you to reduce costs by decreasing your storage footprint while at the same time maintaining the ability to perform analytical queries of your monitoring data over longer time horizons.In this post, we'll cover what downsampling is, and show you how to combine data retention policies and continuous aggregates to save money - while maintaining analytical power of your data.We'll use a hypothetical example to demonstrate how to reduce storage needs from 260,000 rows/day tonine, keeping only the summaries of time-series data, instead of the full-fidelity dataset.What is downsampling?Historical data provides baseline performance context for your applications - by measuring “normal” performance, you can identify and predict anomalous performance. In order to get the most of your historical data, you can leverage downsampling to help you avoid the trade-off of historical data value vs. storage cost and management.Downsampling is the act of applying a mathematical aggregation function (i.e.AVG()) to roll up a very granular time series data set (i.e.  3 second intervals) to a more coarse grained set of data (1 hour, 5 hours, 1 day averages as examples). As a result, your data can take on a new role: analytics.For example, let’s assume we are monitoring a single machine instance, and for the purposes of this exercise we are only focused on CPU usage metrics. We are monitoring a 8 CPU core instance and we measure usage in the system and user spaces, collecting data every 3 seconds.Here is what something like this will look like on a monitoring dashboard:In order to understand what is happening in real time, we need to collect the data at this high velocity and high frequency (Hint: a skill TimescaleDB is built for).However, as this data ages, its purpose changes. When data is new, individual data points are important for debugging and real-time feedback. As data ages, the importance of individual data points often matters less than statistical analysis over large amounts of data.If we downsample historical data, it will still help us spot trends, set baselines for what we consider “normal”, and allow us to be more accurate in our predictions around future behavior. At the same time, downsampling will reduce storage volumes. Let's take a look at how we can make this happen.The downsampling process: a brief tutorialFirst, we need to decide what data to downsample. Let’s assume the data in its original format is collected every three seconds. Let’s say we need a different time-series for analysis purposes (for example to 1 hour and 5 hour and daily averages).We can do this withcontinuous aggregates,time_bucket, andAVG()functions in TimescaleDB to roll up the VERY granular 3 second interval, to views that offer the data at 1 hour, 5 hour, and daily intervals.To help us manage the storage costs, we are also going to use the TimescaleDBdata retention policiesto remove data after the five day window. Let's walk through these steps.#1 Create a continuous aggregateIn this case, we will create acontinuous aggregatewith a daily average of the CPU usage in both the user and system space. We will roll up the 3 second data to a daily average for these metrics. This is what it will look like:CREATE VIEW CPU_daily_rollups
WITH (timescaledb.continuous,
    timescaledb.ignore_invalidation_older_than='5d',
    timescaledb.refresh_lag = '-30m',
    timescaledb.refresh_interval = '1d')
AS
  SELECT time_bucket('1d', time), cpu, AVG(usage_system) AS system_usage, AVG(usage_user) AS user_usage
    FROM cpu
    GROUP BY time_bucket('1d', time), cpu;As you can see, we are building a continuous aggregate that will produce our daily averages using the time bucket function (for more information on building a continuous aggregate clickhere), and we are refreshing this view once per day.The continuous aggregate job that we have created above will  take the average of the utilization across the entire 24 hour period, and rather than needing to keep259,200rows per day (which I needed when I was monitoring this in real-time), I can simply keep 9 entries to represent daily CPU Usage (one per core and a total), which will look like this:SELECT * from cpu_daily_rollups;Now I can simply repeat this process to create continuous aggregates for 1 hour and 5 hour windows and in this use case I will have everything I need for long term analysis.#2 Add a data retention policyThe second part of this exercise is to reclaim the space that is being taken up by the underlying 3 second data points.As I mentioned earlier, we are storing a little more than 259K rows per day per monitored machine in this case. The data has served its purpose for real-time monitoring and has been converted to a less granular form more appropriate for long-term analysis (see above). The next step is to set up a policy that will start to delete the finer-granularity data we originally collected.In this case, we will use a TimescaleDBdata retention policy:SELECT add_drop_chunks_policy('cpu', INTERVAL '5 days', cascade_to_materializations=>FALSE);Here we are dropping the underlying data after 5 days. While we are dropping the granular 3 second data records, we will maintain our continuous aggregate views of this data.#3 Perform analyticsNow that we have created the needed view and downsampled our data, we can start the process of running analytics on that data.To illustrate, I’ve connected an Excel Sheet to my TimescaleDB instances – and set up a basic pivot table that plots the CPU usage To illustrate, I’ve connected an Excel Sheet to my TimescaleDB instances – and set up a basic pivot table that plots the CPU usage we set up in Step 1 (i.e., our continuous aggregate that rolls up our data from three second intervals to hourly averages).Recap & next stepsIn this post, we’ve covered an overview of downsampling and how – and why – it’s important to leverage it for IT monitoring use cases. Of course you can apply continuous aggregates and data retention polices to a variety of other scenarios. If you are interested in learning more about how continuous aggregates work and to see if they are a fit for you, read this blog “Continuous aggregates: faster queries with automatically maintained materialized views”.If you are ready to start downsampling, we encourage you to check out this documentation:Continuous aggregatesData retentionNote: For reference, the ability to enable true downsampling described in this post is included with the recent release ofTimescaleDB 1.6. If you are interested in staying up-to-date with all of our releases, sign up for ourRelease Notes.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-proactively-manage-long-term-data-storage-with-downsampling/
2019-10-17T16:56:00.000Z,Build a Data Pipeline With Apache Kafka and TimescaleDB,"IntroductionIn this post, we will cover how to set up TimescaleDB (hosted onManaged Service for TimescaleDB) as a Kafka consumer (via the JDBC sink connector) with theConfluent Platformfor the streaming data. We will start with a pre-created topic and a schema that has already been defined in the Confluent platform, and we will cover ways to create the schema.Whether you are generating system infrastructure readings (for DevOps / IT monitoring) or capturing edge devices measurements (for IoT), TimescaleDB can be used as a centralized repository for time-series data.To ingest this data into TimescaleDB, our users often create a data pipeline that includesApache Kafka. For those who are not familiar, Kafka is a distributed streaming platform that allows systems to publish data that can be read by a number of different systems in a very resilient and scalable way. A popular way users leverage Kafka is viaConfluentwhich is a comprehensive platform for event streaming.Why Apache Kafka, Confluent and Timescale DB?Kafka isarguablythe most popular open-source, reliable, and scalable streaming messaging platform. It provides a very easy, yet robust way to share data generated up/down stream. The Confluent platform glues together the bits needed for using Kafka and the connector. And TimescaleDB is an open-source database built for analyzing time-series data with the power and convenience of SQL. Essentially, if you are collecting time-series data and looking for an easy streaming messaging platform to handle the routing of that data to TimescaleDB, you should consider this setup.Convinced? Ready to get started?Before you startInstall TimescaleDBFor the purposes of this demonstration, we are going to use TimescaleDB hosted on Managed Service for TimescaleDB. You can sign up forManaged Service for TimescaleDB herefor a 30-day free trial.Install ConfluentNext, you will need to set up Confluent. You can get started with Confluent (if you are not already running the platform) via theirquick start guide.Understand the data pipelineNow, let's take a high-level view of the setup:From the diagram above, you can see we are ingesting data into Kafka from upstream data sources (e.g. servers, edge devices). This data will pass through a Kafka topic that is subscribed to via the Kafka Connect JDBC sink connector, which inserts that data into TimescaleDB for storage and processing.Set up the JDBC sink connectorSince TimescaleDB is built on top PostgreSQL, any tools or extensions that work with PostgreSQL work with TimescaleDB. This means we can use theJDBC sink connectorthat is readily available to make the Kafka-to-TimescaleDB connection.Let's start by setting up our TimescaleDB instance as a consumer of a Kafka topic. For this we are going to use theJDBCSinkConnectoraccessed through the Confluent UI.Here we are accessing the Kafka Connect functions in Confluent to setup the JDBC sink connector, and subscribe our TimescaleDB instance to a topic. The next set of configuration steps are form driven in the Confluent platform. The platform provides tooltips for each setting, in the next few paragraphs we will cover the most relevant items.The first few sections of the setup form are straightforward. First we are going toassociate the connection with a topicor set of topics (you will want to have a topic set up at this point).The next field is preselected for you and it askshow you want Kafka to connect to your data(in this case JDBC sink connector is pre-selected). Next, we want togive our connection a name.The following section of the setup process covers somecommon settings. For the purposes of this demo, we were able to accept the defaults, however your setup might require some configuration in this section. Let's quickly go over the settings:Tasks max:This will refer to the max number parallel tasks the connector will attempt to perform.Key converter class:This will be setup used to convert between the Kafka Connect and the serialized form that is written to Kafka (Message Keys). Settings here would be driven by how upstream data is brought in. Popular options are JSON and Avro.Topics regex:This can be filled out if you want to associate this connection with a series of topics. (Using the regex saves you time by listing them all and will ensure future topics that match the regex are subscribed to).Value converter class:Similar to key converter class, only this is the message value converter. Like key converter class it will be driven by how upstream data is brought into the topic.Header converter class:Similar to value and key converter class this instance is dealing with the message header.Reload action:The action that connect should take when changes in external configuration providers result in changes in the connector configuration.Transforms:Here you can select any transformations you may have defined (a list of supported transformation types ishere).The last setting that’s important to note iserror handling. Most are self explanatory, but lets cover two of the more important items.Retry timeout for errors:This is by default zero which means no operation will be retried on error. This should be set in accordance with your environment's error handling policy.Error tolerance:This will happen when there is a message error. The default is to stop the connector processing, and setting this to “All” will result in skipping the problematic record and continuing.Configure connection parametersAfter you determine your settings, it’s time to connect to TimescaleDB:Connection:JDBC URL:This is straightforward and should look something like this:jdbc:postgresql://<hostname>:<port>/<database>JDBC User:User for connection.JDBC Password:Password for connection.Database Dialect:This will be inferred by the JDBC connection string, however this can serve as an override.Writes:Insert Mode:How you want to see messages processed into the database? In the case of time series data, we are going to define INSERT as the default, as this will be our primary operation (vs. update).Batch Size:How many records do you want to batch together for insertion into the destination table?You can obtain your TimescaleDB instance connection information from the Managed Service for TimescaleDB portal.Configure data mappingThe next set of fields tell the connector about how you would like the data mapped once it lands in TimescaleDB. Let’s have a look at the fields:Table Name Format:This allows you to define table name convention (will also be used to check if table already exists).  For example, if you wanted to create a naming convention that called the tableskafka_<topic_name>you could define this by enteringkafka_$(topic)Fields Whitelist:List of comma separated field names to be used. If left empty, it will use all fields.DB Time Zone:Name the JDBC timezone uses for timestamp related data.The JDBC sink connector will auto create the table that it needs, and will also auto evolve the table if the schema changes.Auto-Create:Will auto create the table if it does not exist.Auto-Evolve:Will alter the table automatically if the schema should change.We recommend setting Auto-Create to false.If you would like to save some keystrokes, you can allow the Auto-Create to create the base table in your test environment, stop the data flow, and truncate the table and convert the table to a hypertable. Once that is done, you can turn the data flow back on. If you are going to create the table yourself based on the message schema, be sure to follow the naming convention (by default it will look to name the table based on the topic name) so the connector finds the table when the sink connector is turned on.We recommend setting Auto-Evolve to false.The schema should be considered end-to-end and a change made upstream that is pushed downstream may create performance issues.Set up TimescaleDBNext, we need to configure the database to ingest the data from the JDBC sink connector (from Kafka). We will now need to create the table that will receive the data from the Kafka topic by way of the JDBC sink connector. To do this you have two options: leverage auto create or manually create the table.Option 1: Leverage Auto CreateAs part of the JDBC sink connector process, you have the ability to let the connector create the table. While there is not an option to allow this to create ahypertable, you can turn on the connector and allow it to generate the table structure for you. Once this is done, turn the connector back off, truncate your table, and follow the instructionshereto convert the table to a hypertable.Once this is completed, you will be ready to turn the connector back on and allow the data to flow from the JDBC connector into your newly created hypertable.Option 2: Manually create the tableThe other option is to manually create the table based on your knowledge of the topic schema. Here you will want to make sure you name the table based on the topic name (accounting for any changes to the default you may have set while setting up the JDBC sink connector). Once you have created the table, you can follow the same instructionshereto convert this to a hypertable.After you have created the hypertable, and assuming you have your JDBC sink connector working and associated with a working topic, you can now turn on the JDBC sink connector and you should start to see data flow from the Kafka topic through the connector and into the TimescaleDB hypertable.Another use case: data generationSo far, we’ve assumed that you are preparing to ingest data from an existing topic, or a topic that you will make available as you bring TimescaleDB online. If for some reason you are simply looking to do a PoC or other test that would bear out the data pipeline, the Confluent platform also includes a data generation connector. That connector supports a number ofpredefined schemasthat are supported by the random data generator for use in generating random data that is published to a Kafka topic, then pushed to the JDBC connector, and lands in the TimescaleDB hypertable you have configured.This is a great option if you just want to run a small proof of concept, or would like to do some stress testing with the entire stack that you have set up. Confluent has a number ofquick start guidesthat will help you generate data quickly.Next stepsWe hope this post helped you get TimescaleDB up and running with Kafka and Confluent. As a reminder, if you are looking to connect a technology with TimescaleDB that has an existing connection to Postgres, it will work!We encourage you to visit ourManaged Service for TimescaleDB documentationpage andsign up for a free trial. If you prefer to deploy on-premises, you can explore the available installation optionshere.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/create-a-data-pipeline-with-timescaledb-and-kafka/
2021-08-03T12:10:53.000Z,"Move Fast, but Don’t Break Things: Introducing the Experimental Schema (With New Experimental Features) in TimescaleDB 2.4","At Timescale, we have a track record of constantinnovation, moving quickly to solve painful problems that developers who work with time-series data experience (For example, we recentlyshipped 12 launches in a single month!)But on the other hand, we also valuestability– and so do our users. We cannot change our API every time we release a new database version, nor can we expect TimescaleDB users to constantly make changes to their production code in order to keep their database working, or risk breakages if they don’t. The potential damage caused by “moving fast and breaking things” could alienate the very users who rely on our product: developers who expect stability and reliability from their database.At Timescale, we build critical infrastructure components, and our users expect and rely on stability (see our aptly titled“when boring is awesome” launch announcement). As the makers of TimescaleDB, the leading relational database for time-series data, we recognize that there are nearly three million active TimescaleDB databases running mission-critical time-series workloads across industries. Our prime directives are to safeguard data and ensure our customers’ applications do not need to be rewritten with every release, and to continuously build features big and small to make developers’ lives easier.We made the decision early in the design of TimescaleDB to build on top of PostgreSQL. We believed then, as we do now, that building on theworld’s fastest-growing databasewould have numerous benefits for our customers. Among these benefits is the rock-solid dependability developers have come to expect from a database with 20+ years of production usage behind it. Building on top of PostgreSQL enables us to innovate rapidly with solutions for ingesting and analyzing time-series data while also leveraging (and contributing to) a database with a reputation for stability.Striking a balance between innovation and stability is not a problem unique to TimescaleDB, but it’s perhaps the most important question in our software development process. We strive to achieve the dependability developers expect from PostgreSQL, yet also aim to deliver rapid innovation and progress for customers building solutions in an industry that constantly changes.So, how do we resolve the tension between innovation and stability so that we can continue to ship new functionality quickly while retaining the trust we’ve earned with our customers?Today, to reinforce our commitment to moving fast andnotbreaking things,we are introducinga new experimental schema for TimescaleDBas part of our release of TimescaleDB 2.4.The experimentalschemais where we aim to develop and ship features at an even faster pace than we normally do. The general objective is that these new functions will “graduate” out of the experimental schema when they reach full maturity for normal production usage.TimescaleDB 2.4 introduces the following experimental functionality in the new experimental schema:A function to analyze data in arbitrary time intervals that improves on the populartime_bucketfunction, with support for bucketing using months and years and support for time-zone use incontinuous aggregates. Youasked, and we delivered.Functionality for elasticity and high availability in multi-node TimescaleDB. Specifically, functions to copy or move data (at the chunk level) between data nodes.Read on for more about how we introduced experimental functionality in TimescaleDB, details about new experimental functionality, and examples of when and why to use them.TimescaleDB 2.4 is available today for Timescale and self-managed users and will be available in the coming weeks for Managed Service for TimescaleDB users.If you’re new to TimescaleDB and want to try out our new experimental functionality right away,create a free accountto get started (30-day trial 🔥).Join ourSlack communityto share your results, ask questions, get advice, and connect with 7K+ other developers (our co-founders, engineers, and passionate community members are active on all channels).Visit our GitHubto learn more (and, as always, ⭐️ are appreciated!). And, if these are the types of challenges you’d like to help solve,we are hiring!Shoutout to all the engineers who worked on these features: Mats Kindahl, Erik Nordström, Nikhil Sontakke, Aleksander Alekseev, Dmitry Simonenko, and the entire team of reviewers and testers!We’d like to give a special shoutout to all community members who’veasked for improvementsto thetime_bucketfunction and participated in ourdiscussions on GitHub, both of which informed our choice of experimental functionalities introduced today.Introducing the experimental schema in TimescaleDBExperimental schemas enable us to validate new functionality and get feedback from real users, and enable developers to test the latest and greatest of what TimescaleDB has to offer.Moreover, the schema’s explicit experimental name and the “experimental” component of API functions act as a “buyer beware” label and set expectations, telling users to anticipate breaking changes and to not use these features in production until they’ve “graduated” and are stable. This frees our team to make changes to existing features without breaking compatibility with functionality that users rely on and mature features through iterative improvements until they are ready for wider consumption.The experimental schema is an important milestone for our users and us, demonstrating Timescale’s commitment to innovation, not just in the features and products we release but in the process of how we actually build and deliver our products to delight developers. It also gives us a sandbox where users can test the latest features and more actively shape the development of TimescaleDB.We first released experimental functionality inTimescaleDB hyperfunctions,a series of SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in PostgreSQL with fewer lines of code.In addition to production-ready hyperfunctions for calculatingtime-weighted averagesandpercentile approximations, we released several hyperfunctions in public preview for developers to trial through our hyperfunctions experimental schema and offer feedback to improve the features so that they may be released for production usage soon. These experimental hyperfunctions included functions for downsampling, smoothing, approximate count-distinct, working with counters, and working with more advanced forms of averaging.How experimental features workAs noted previously, functionality under the experimental schema isexperimentalin nature and not recommended for production use at this time. You should expect that several of these functions may change their APIs, change their names, change views, or disappear. Additionally, when experimental features “graduate” from the experimental schema, they’re not guaranteed to allow moment-in-time upgrades.Here’s how features graduate from the experimental schema to the main schema (and are thus deemed ready for production use):Step 1: We release Feature X in the experimental schema, tell users about it, and gather user feedback. Users give feedback bysubmitting new issues or commenting on existing issues in GitHub.Step 2: Depending on the amount and severity of issues uncovered during the feedback process, we spend time addressing the feedback. This can take weeks to months, and would be one or two minor versions after the experimental version was released (i.e., if the experimental version was released in v2.4, the production-ready version might be ready to use in v2.5 or v.2.6)Step 3: Once we’ve stopped receiving bug reports, we graduate Feature X from experimental to production-ready and notify users in our release notes. On the other hand, if we continue to receive bug reports about Feature X while it is in the experimental schema, we keep it in the experimental schema until we stop receiving bug reports (and the feature passes our own internal QA) and then graduate it in the next available release.Ultimately, customer feedback drives the pace at which features graduate through each step and into production.time_bucket_ng: An upgrade to time_bucketThe first new experimental feature we are releasing today is one of ourmost requested featuresand tops many of our community members' wishlists: an improved version oftime_bucket, with support for month, year, and time zones. Thanks toJean-Francois Labbéfor getting the ball rolling 3 years ago:AGitHub issuefrom TimescaleDB community member Jean-Francois Labbé made in January 2018 asking for support for monthly intervals intime_bucket.Background on time_bucketFor those new to TimescaleDB,time_bucketis among the most popular TimescaleDB hyperfunctions and is used for bucketing and analyzing data for arbitrary time intervals in SQL. For readers familiar with PostgreSQL, you can think oftime_bucketas a more powerful version of thePostgreSQLdate_truncfunction.time_bucketallows for arbitrary time intervals, rather than the standard day, minute, hour provided bydate_trunc, enabling more flexibility in doing analysis using the time periods which most matter to your use case.For example, you could usetime_bucketto find the average temperature in 15-day buckets for a certain city, using the following simple query:-- time_bucket
-- Average temp per 15 day period
-- for past 6 months, per city
-----------------------------------
SELECT time_bucket('15 days', time) as ""bucket""
   ,city_name, avg(temp_c)
   FROM weather_metrics
   WHERE time > now() - (6* INTERVAL '1 month')
   GROUP BY bucket, city_name
   ORDER BY bucket DESC;What’s currently missing in time_bucketWe love hearing from our customers and community members. Our entire team regularly reads and answers questions in ourpublic TimescaleDB Slack community. Our Eon Bot (named after our mascot, Eon) helps us identify long-requested functionality, inviting members of our community to vote on feature requests.We also opened up ourGitHub discussionsas a community exercise on what we should scope related to thetime_bucketfeature. The most voted solution was:I would like to specify a timezone like ""UTC+3"" for a given continuous aggregate so that any time from a source table will be converted to the given timezone and end up in a right bucket (in terms of when the next day starts, etc). Multiple continuous aggregates can be created, each with its own time zone, e.g “UTC” one and “UTC+3” one. I also would like to use timezones like ""Europe/Berlin"". Unlike simple “UTC+3”, such timezones account for daylight saving time and will automatically change time. This means that there can be days with 23 or 25 hours, which is not the case when specifying a specific time zone like CEST and CET.The Eon Bot and GitHub discussions are two examples of how to incorporate community feedback in our feature development process.As you can see, the most common requests we received aretime_bucketrelated, specifically: support for months/days/years intime_bucket, support for timezones intime_bucket, and being able to use both in continuous aggregates. The original issues are as old as three years old, with more than 65 upvotes on the original comment. Although providing the requested functionality may sound simple, it’s not trivial to implement it.There are several reasons for this, but the most important one is that thetime_bucketfunction and continuous aggregates were originally implemented for buckets that are fixed in size, like 10 seconds, 3 hours, or 5 days. However, months and years are variable in size: months range between 28 and 31 days, years range between 365 and 366 days. Breaking this constraint requires many changes throughout a large part of the TimescaleDB code base, and you have to keep backward compatibility and performance in mind. All in all, this is doable, but it takes time.Time zone support is also a case of variable-sized buckets: If there is adaylight saving timechange in a given timezone, it means there are days that have between 23 and 25 hours.We had to re-engineertime_bucketfor these kinds of variable-sized buckets.The next generation of time_bucketEntertime_bucket_ng(NextGeneration) with implemented support for: Years, Months, Weeks, Days, Hours, Minutes, and Seconds.time_bucket_ngalso includes support for use with continuous aggregates. The new support for years and months enables a host of capabilities for analysis that looks over longer time periods (e.g., business dashboards showing month-over-month growth).To make this a bit more concrete, we’ve included a few examples oftime_bucket_ngin action.Here’s how to usetime_bucket_ngto create bucket data in 3-month intervals:-- for our readers from North America: the date is in YYYY-MM-DD format
SELECT timescaledb_experimental.time_bucket_ng('3 month', date '2021-08-05');
 time_bucket_ng
----------------
 2021-07-01
(1 row)Here’s how to usetime_bucket_ngto bucket data in 1-year intervals:SELECT timescaledb_experimental.time_bucket_ng('1 year', date '2021-08-05');
 time_bucket_ng
----------------
 2021-01-01
(1 row)Here’s how to usetime_bucket_ngwhen creating continuous aggregates. In this case, we track the temperature in Moscow over 7-day intervals:CREATE TABLE conditions(
  day DATE NOT NULL,
  city text NOT NULL,
  temperature INT NOT NULL);

SELECT create_hypertable(
  'conditions', 'day',
  chunk_time_interval => INTERVAL '1 day'
);

INSERT INTO conditions (day, city, temperature) VALUES
  ('2021-06-14', 'Moscow', 26),
  ('2021-06-15', 'Moscow', 22),
  ('2021-06-16', 'Moscow', 24),
  ('2021-06-17', 'Moscow', 24),
  ('2021-06-18', 'Moscow', 27),
  ('2021-06-19', 'Moscow', 28),
  ('2021-06-20', 'Moscow', 30),
  ('2021-06-21', 'Moscow', 31),
  ('2021-06-22', 'Moscow', 34),
  ('2021-06-23', 'Moscow', 34),
  ('2021-06-24', 'Moscow', 34),
  ('2021-06-25', 'Moscow', 32),
  ('2021-06-26', 'Moscow', 32),
  ('2021-06-27', 'Moscow', 31);

CREATE MATERIALIZED VIEW conditions_summary_weekly
WITH (timescaledb.continuous) AS
SELECT city,
   	timescaledb_experimental.time_bucket_ng('7 days', day) AS bucket,
   	MIN(temperature),
   	MAX(temperature)
FROM conditions
GROUP BY city, bucket;

SELECT to_char(bucket, 'YYYY-MM-DD'), city, min, max
FROM conditions_summary_weekly
ORDER BY bucket;

  to_char   |  city  | min | max
------------+--------+-----+-----
 2021-06-12 | Moscow |  22 |  27
 2021-06-19 | Moscow |  28 |  34
 2021-06-26 | Moscow |  31 |  32
(3 rows)(Seeour continuous aggregates docsfor more information about continuous aggregates, and check out ourgetting started guidefor tips on how to use them and other TimescaleDB advanced features.)time_bucket_ngdemonstrates the power of using the experimental schema to solve long-standing community requests without introducing breaking changes. It also gives community members the opportunity to give feedback about the new functionality, so we make sure our implementation solves real users’ problems. (Please let us know the results of your testing and new functionality you’d like on ourGitHub issues page.)In the next iterations oftime_bucket_ng, we will release support for: 1 month in continuous aggregates; N-months and N-Years with continuous aggregates; and time zones.The following chart indicates the differences intime_bucketfunctionality:time_bucket_ngadds support for bucketing by years and months and timezone functionality for use in queries, as well as in continuous aggregates.Introducing improved elasticity and high availability for multi-node TimescaleDBWe introducedmulti-node TimescaleDBin TimescaleDB 2.0, and with it, the ability for users to run petabyte-scale workloads across multiple physical TimescaleDB instances (called data nodes). Multi-node TimescaleDB enables horizontal scaling and rapid ingest of the most demanding time-series workloads and was designed from the start to be highly available and elastically expand as data needs grow.However, an important piece of the puzzle has been missing, namely the ability to move and copy data around the cluster (collectively, all the nodes in your deployment). Let’s look at elasticity and high availability separately to see how copying and moving data helps in those situations.Elastically scaling up and downIt is already possible to horizontally expand your multi-node TimescaleDB database by adding a new data node on self-managed TimescaleDB deployments. However, this doesn’t provide immediate value: a new node will join the cluster without any data and will neither serve data nor alleviate pressure on storage or compute for existing nodes. Thus, you need to wait until enough new data is written to the new data node before it’s used in practice.In TimescaleDB 2.4, we introduce a new experimental function (strictly aPostgreSQL procedure) that frees you to move chunks from existing data nodes to new data nodes:CALL timescaledb_experimental.move_chunk(‘_timescaledb_internal._dist_hyper_1_1_chunk’, ‘data_node_2’, ‘data_node_3’);For example, this makes it possible to add a new data node and move old chunks to the new data node to alleviate storage pressure.Conversely, it is also possible to delete a data node, but, in order to preserve the integrity of the database, it is not possible to do it as long as the node still holds data. By moving chunks off the data node, it is now possible to take a data-serving node out of rotation, either to replace it with a differently configured node or simply to scale down the cluster in order to reduce costs.TimescaleDB already includes amove_chunkfunction that allows users to move chunks across tablespaces (i.e., disks). Our first approach was to extend this existing function to work acrossnodes.However, moving a chunk across two data nodes is a multi-transactional operation, and standard PostgreSQL functions can’t run multiple transactions. Therefore, we had to use aPostgreSQL procedureinstead of a function, which would introduce a breaking change to the currentmove_chunkfunction.With the experimental schema, we can now introduce multi-node scaling much faster for users who’d like to try it out and continue to improve the functionality, without impacting existing interfaces for production use cases. In the future, we might deprecate the existing move function in favor of a unified interface.High availabilityMulti-node TimescaleDB already supports high availability, achieved by usingPostgreSQL’s streaming replicationto add standby(s) to each node in the cluster. However, for a large multi-node cluster, this can be a management-intensive and costly solution where the extra resources aren’t optimally used.As an alternative to the traditional standby setup, multi-node TimescaleDB offers built-in functionality to replicate data at the chunk level. This allows each data node to work both as a primary for some chunks and backup for others. If a data node fails, its chunks already exist on other nodes that can take over the responsibility of serving them.However, after a node failure, there will be some chunks that no longer have the desired number of replicas to sustain further failures, i.e., those chunks are under-replicated. In order to get back to a fully replicated multi-node cluster, an under-replicated chunk needs to be copied to a data node that doesn’t already have a copy. The target node can either be an existing data node or a newly added one.The experimental schema offers a copy function, which is similar to the move function:CALL timescaledb_experimental.copy_chunk(‘_timescaledb_internal._dist_hyper_1_1_chunk’, ‘data_node_2’, ‘data_node_3’);Failures when copying and moving chunksA copy or move operation for a chunk might fail or be interrupted, and in that case, the incomplete operation has to be rolled back.However, as mentioned above, a move or copy happens over multiple transactions so the operation cannot be completely rolled back by PostgreSQL. Instead, TimescaleDB tracks each step of a move or copy, so failed operations can be undone (or resumed).In the experimental schema we have introduced a function to clean up a failed copy or move:CALL timescaledb_experimental.cleanup_copy_chunk_operation('ts_copy_1_31');The function needs to be passed the operation ID that will be logged in case a copy or move operation fails.Provide feedback and help drive the projectWe invite – and encourage – all community members to try the experimental schema in a test-safe environment and to tell us about your experience.What should we improve?Were things easy to understand, or was something difficult to set up or tricky to use?How did you expect a feature would work and did it meet (or exceed!) your expectations?We want to hear it all: let us know about the glitches you encountered, anything that surprised and delighted you, and everything in between.To share feedback,create a GitHub issue(using theexperimental-schemalabel), describe what you found, and tell us the steps - or share a code snippet - to recreate it.We’re committed to developing these and future experimental features – and to giving the community a chance to provide early feedback and influence the direction of TimescaleDB’s development. We’ll travel faster with your input and we truly appreciate your contribution(s) (thank you in advance 🙌!).Get started todayIf you’re new to Timescale and want to get started with our new experimental functionality today,you can self-host to get started. If you want to try our engineeredPostgreSQL cloud platform, try our 30-day free trial(no credit card required).If you are an existing user:Timescale:TimescaleDB 2.4 is now the default for all new services on Timescale, and any of your existing services will be automatically upgraded during your next scheduled maintenance.Managed Service for TimescaleDB:TimescaleDB 2.4 will come in the next few weeks.Self-managed TimescaleDB:Here are upgrade instructionsto upgrade to TimescaleDB 2.4.As a reminder, you can join ourSlack communityto share your results, ask questions, get advice, and connect with other developers (our co-founders, engineers, and passionate community members are active on all channels).You can alsovisit our GitHubto learn more (and, as always, ⭐️  are appreciated!) And, if these are the types of challenges you’d like to help solve,we are hiring!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/move-fast-but-dont-break-things-introducing-the-experimental-schema-with-new-experimental-features-in-timescaledb-2-4/
2019-08-15T18:26:50.000Z,"Build an Application Monitoring Stack With TimescaleDB, Telegraf & Grafana","Match the flexibility and scale of your application with a stack that works for you.The world of systems design has become a powerful yet complex place. Through the advancement of microservice architectures, enterprise applications have become more fault tolerant, easier to scale, and capable of delivering end users with a constantly improving experience due to the ability for development teams to rapidly iterate and innovate.However, this presents the operations teams with some VERY complex challenges when it comes to monitoring the health and performance of these applications. For example:How do we implement a monitoring solution that matches the power and flexibility of the application we are deploying?How can we have an eye on all application layers and collect all the data we need to monitor key metrics that ensure the end user is having an optimal experience?The answer to those questions are simpler than you might think. We must look to anapplication monitoringstack that matches the flexibility and scale of the application we are monitoring.This monitoring stack can use best of breed components to instrument the needed data collection, and to store the data in such a way that is can be accessed quickly and can be used for both realtime and historical contexts. The end result provides us the ability to present the data based on the needs of the teams maintaining the application.In this blog, we’ll discuss such a stack using the following:Application Monitoring StackTimescaleDBTo collect the time series data that will be generated by as part of our monitoring, and provide a highly scalable and performant platform to store this data and make it available for both realtime and historical analysis.TelegrafTo instrument the data collection across the application.GrafanaTo provide the requisite needed visualization of the data points collected in TimescaleDB and allow us a window into the data being served by Timescale.Storing the data in TimescaleDBNow that you know what this application monitoring stack will look like, the first order of business is to make sure you have a TimescaleDB instance running (whether on premise or an instance running inManaged Service for TimescaleDB. TimescaleDB is the heart of the application monitoring stack, and is where the data from your application will land.If you are brand new to Timescale, follow the instructionshereto get TimescaleDB going locally or in the cloud.The nature of the data we are collecting is unique, as you can see from the sample below it represents time-series data:In the sample, we are capturing CPU metrics on a minute by minute basis, calling for a database technology that is purpose built to handle this type of data ingestion (large volume, high velocity), and obtains the requirements for querying this type of data. TimescaleDB is designed to help manage this type of data ingestion and complex queries.In our application monitoring use case we are going to look to cast the data in two ways:We need to be able to query the data in such a way that it will allow us to build real time dashboards, helping us understand what is happening “now”.We need to be able to store and query the data in a historical context, that is to say we need to be able to understand, and start to predict what will happen in the future (allowing us to prepare and/or budget for additional resources, and make the needed application level adjustments).Being able to serve these two use cases simultaneously is key since it represents the core of our application performance monitoring stack, and is at the core of the value provided by TimescaleDB.Another large benefit in using a technology like TimescaleDB is that it sits on top of PostgreSQL, making it simple to get the data out of the database. Whether you're using a tool like Grafana to build a dashboard (which we will discuss next) or want to run some ad hoc queries to understand trending in storage usage, the idea that you will be using standard SQL to do this reduces the learning curve and shortens the time to value of the entire stack.Instrumenting data collection with TelegrafThe next step is figuring out how we are going to collect data, and in this case we are going to use Telegraf and deploy it to all elements of our application.Telegraf gives us the facility to collect the information we need to properly monitor the health and performance of our application. We will be able to gather things like CPU, Memory, Network, as the basics (i.e. will operate at the pod level in a Kubernetes environment, and report pod based statistics if you have chosen this type of deployment). We can also instrument data collection / metrics that will help us evaluate things like database performance and application response times.The Telegraf agent is lightweight and simple to install, and will solve the first part of our problem: data collection.Using the linkhereyou will find step by step instructions for deploying Telegraf (Note: This version of Telegraf includes output plugin support for writing back to PostgreSQL and TimescaleDB).Configuring visualization & alerting with GrafanaFinally, let's talk about visualization and about how are we going to present the data we are collecting and use it to monitor what is happening across the application.In this case, we are going to use Grafana to help us understand the data in realtime while giving us the ability to set and trigger alarms when something is out of specification. (To set up Grafana visualization with TimescaleDB, follow theseinstructions.)As an example, we may choose to collect CPU data from the nodes in our application cluster, and we will want to make sure we monitor this in real time and set alarm thresholds to notify of a potential issue:We are capturing the real time data from the machine, monitoring both System and User CPU usage, and setting an alarm threshold when we reach 80% utilization. This is an example of data being evaluated and shown to the user in real-time. This particular dashboard is updated every 5 seconds.In contrast, we also need a higher level view of the same world. In the case below we are looking at the same set of CPU metrics but across a broader period of time (12 Hours):Again, being able to look at the larger picture as it relates to spotting trends is another use case we need to account for in this solution. The ability to pull back from the granular level, and view historical data is key to managing our application and its performance.In this case we are looking for spikes based on a particular window of our day (12 hours), however it is worth noting that because TimescaleDB is providing us with long term data storage we can pull back to a daily, weekly, or even monthly views of this data, with an eye on being proactive in spotting the trends associated with our application performance.Once this data is integrated into Grafana, we can define alert rules (e.g. “Average CPU usage greater than 80 percent for 5 minutes”). Once an alert is triggered, Grafana can dispatch a notification. (Instructions for setting up alerting in Grafana can be foundhere.)SummaryWhen it comes to implementing and application performance monitoring stack, flexibility along with best of breed components that are purpose built to carry out their part of the job is the key. Being able to understand the real time “situation” AND quickly and easily access historical trends is required.Our suggested stack will give you the flexibility to collect that data you are interested in, store it in a database that will serve the two key use cases (real-time and historical), provide you the flexibility around how you want to display the data, and ensure the data is collected and queried in a performant manner. All of this will make certain that you have access to the data you need to effectively manage your application.Have questions about setting up thisapplication monitoringstack? Reach out to us on our communitySlackchannel.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/build-an-application-monitoring-stack-with-timescaledb-telegraf-grafana/
2019-11-26T16:13:48.000Z,Recommendations for Setting Up Your Architecture With AWS & TimescaleDB,"Learn more about common implementations and find the option that works best for your project, team, or organizationWithAWS Re:Inventright around the corner, we thought it might be a good time to re-cap how you can use AWS services in conjunction with TimescaleDB.We find that a lot of our users implement an architecture that uses AWS services, and add TimescaleDB to their stack to manage, store, and analyze all their time-series data at scale. Fortunately, since TimescaleDB seamlessly integrates with many AWS offerings, there are several ways to create a flexible stack that works for you.In this post, we will discuss different ways to set up your architecture, and provide recommendations to help you choose the best option for your use case.Option #1: Timescale CloudThe first option that comes to mind is Timescale’s managed service offering,Timescale Cloud. This is a path allows you to host your TimescaleDB instance on AWS and use Virtual Private Cloud (VPC) peering to connect the instance to the rest of your AWS infrastructure.The primary benefit to the managed service is that you can be hands off in terms of the day-to-day management of the system, since we manage updates and upgrades - along with backups and high-availability (HA). With this route, you also get the flexibility to customize compute and storage configurations based on your needs, and grow, shrink, or migrate your workloads with just a few clicks.Here is an overview of what this setup would look like:If you are interested in running TimescaleDB as a managed service on AWS, this is the option to explore. This configuration will offer you the benefits of the managed service, while running on the AWS platform, and allow you to add it to the rest of your cloud stack through VPC Peering.Summary:Best for users who prefer a hands-off approach and need flexibility when it comes to managing their data.Not ideal for users who want to be heavily involved in the day-to-day management, or who need granular control over the environment.Option #2: EC2 InstanceIn the event you’re looking for more granular control over your instance and/or more granular control over how the instance runs, you can always spin up an EC2 instance directly on AWS with TimescaleDB preinstalled.To help facilitate this setup, there are a number of community Amazon Machine Images (AMI) to help you get up and running quickly, and we cover this option in ourinstallation docs.A quick search of the Community AMIs for Timescale will get you the results below:(See ourAMI installation docsfor step-by-step instructions)Using the AMI can make it a little easier and faster to spin up an instance, since the AMI is preset with TimescaleDB already installed. Alternatively, you can spin up your own EC2 profile and install TimescaleDB within a custom/tailored environment. As an example, if you have specific requirements around the operating system, or the configuration of the virtual machine (e.g.,. Security software etc.) you will want to go with a more generic image, and customize your installation vs. going with a pre-built AMI.When you select this option, you gain operational control over your instance -- but assume a level of operational responsibility for ongoing maintenance vs. a managed cloud offering.Summary:Best for users who want a lot of control over how their instance is configured and run.Not ideal for users who don’t want to be heavily involved with setup and ongoing management.Option #3: AWS Elastic Kubernetes Service via Helm ChartsThe third option is to deploy Timescale via Kubernetes, using the AWS Elastic Kubernetes Service (EKS). Here, Timescale offers a set ofHelm Charts(freely available via our GitHub repository) to help you facilitate this deployment.Here is an overview of what this setup would look like:This option gives you the ability to deploy TimescaleDB as a cloud-native application, adding the time-series database functionality to your microservices deployment.Summary:Best for users that have embraced and are using a microservices architecture.Not ideal for users that leverage legacy deployment models.Option #4: Amazon Elastic Container ServiceIn the event that you would like to run via the Amazon Elastic Container Service (ECS), Timescale offers aDocker imageto get you started.To illustrate how this works in a real-world scenario, let’s take a look at how we might deploy this in production. Say we want to collect monitoring data from Prometheus and store it in TimescaleDB to identify trends and make future predictions.Here is an overview of what this setup would look like:Here, we’re using Prometheus to monitor a cluster running on-premises (single or multiple locations), and as we collect metrics, they’re written to theTimescaleDB/Prometheus adapter.In this example, we set up the adapter as a container in ECS in order to give this small, yet critical, part of the monitoring stack the availability it requires.After setting up the adapter as a container in ECS, we connect to Timescale Cloud via VPC (our instance runs on AWS and also includes our AWS-hosted Grafana instance). This allows ECS to ensure high availability of the TimescaleDB/Prometheus adapter, while Timescale Cloud, manages the availability of our TimescaleDB and Grafana instance.To learn more about this type of deployment - and why it’s valuable in DevOps scenarios - check out ourHow to use DevOps Monitoring Data to Improve System Stabilityvideo.(TimescaleDB also offers apre-built Docker image with the Prometheusadapter installed. You simply add it to your container registry and deploy it from there.)Summary:Best for users looking to set up a simple, yet powerful application monitoring stack.Not ideal for users that aren’t collecting a lot of monitoring data and don’t need to leverage Prometheus.Option #5: AWS CloudWatch, Lambda, and TimescaleDBTo add to the use case above, a lot of our users combine on-premises or private cloud resources and AWS cloud resources. So, let's talk about monitoring dataconsolidation.In our previous example, we collect metrics from Prometheus directly from our on-premises assets, leaving us with the question: how can I correlate that data with my cloud-based monitoring data?The answer: use an AWS CloudWatch Subscription Filter to send events directly to an AWS Lambda function, which then writes the even to our Timescale Cloud instance (included in our previous example).Here is an overview of what this setup would look like:We’re now storing and analyzing log events, metrics, and/or traces from on-premises and cloud.For another example of how to combine AWS Lambda and Timescale Cloud, check outUsing AWS Lambda with Timescale Cloud for IoT Data.Summary:Best for users who are operating production environments or who’d like to consolidate monitoring data in a single place.Not ideal for users who aren’t collecting a lot of monitoring data yet.Next stepsWhile this is not a complete list of ways you can use TimescaleDB and AWS services, we’ve covered a majority of common use cases (and their high level implementations) to help you navigate your options.Brand new to Timescale?Sign up for aTimescale Cloud account(includes $300 in cloud credits to get you started) or view all available installation optionshere.Attending AWS Re:Invent in Las Vegas next week (Dec. 2-6)?We’d love to see you! Swing by booth #3722 to say hi -- or shoot us an email and we’ll set up dedicated time to meet ([email protected]).As always, we encourage you to join our CommunitySlackchannel to see chat with the team, ask questions, and see what others are working on.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/recommendations-for-setting-up-your-architecture-with-aws-timescaledb/
2021-08-04T12:37:16.000Z,How PostgreSQL Aggregation Works and How It Inspired Our Hyperfunctions’ Design,"Get a primer on PostgreSQL aggregation, how PostgreSQL’s implementation inspired us as we built TimescaleDB hyperfunctions and its integrations with advanced TimescaleDB features – and what this means for developers.At Timescale, our goal is to always focus on the developer experience, and we take great care to design our products and APIs to be developer-friendly. We believe that when our products are easy to use and accessible to a wide range of developers, we enable them to solve a breadth of different problems – and thus build solutions that solve big problems.This focus on developer experience is why we made the decisionearly in the design of TimescaleDB to build on top of PostgreSQL. We believed then, as we do now, that building onthe world’s fastest-growing databasewould have numerous benefits for our users.Perhaps the biggest of these advantages is developer productivity: developers can use the tools and frameworks they know and love and bring all of their SQL skills and expertise.Today, there are nearly three million active TimescaleDB databases running mission-critical time-series workloads across industries. Time-series data comes at you fast, sometimes generating millions of data points per second (read more about time-series data). Because of this volume and rate of information, time-series data is complex to query and analyze. We built TimescaleDB as a purpose-built relational database for time-series to reduce that complexity so that developers can focus on their applications.So, we’re built with developer experience at our core, and we’ve continually released functionality to further this aim, includingcontinuous aggregates, user-defined actions, informational views, and most recently,TimescaleDB hyperfunctions: a series of SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in PostgreSQL with fewer lines of code.To ensure we stay focused on developer experience as we plan new hyperfunctions features, we established a set of “design constraints” that guide our development decisions. Adhering to these guidelines ensures our APIs:Work within the SQL language (no new syntax, just functions and aggregates)Intuitive for new and experienced SQL usersUseful for just a few rows of data and high-performance with billions of rowsPlay nicely with all TimescaleDB features, and ideally, makes them *more* useful to usersMake fundamental things simple to make more advanced analyses possibleWhat does this look like in practice? In this post, I explain how these constraints led us to adopt two-step aggregation throughout TimescaleDB hyperfunctions, how two-step aggregates interact with other TimescaleDB features, and how PostgreSQL's internal aggregation API influenced our implementation.When we talk about two-step aggregation, we mean the following calling convention:Where we have an inner aggregate call:And an outer accessor call:We chose this design pattern over the more common - and seemingly simpler - one-step aggregation approach, in which a single function encapsulates the behavior of both the inner aggregate and outer accessor:Read on for more on why the one-step aggregate approach quickly breaks down as you start doing more complex things (like composingfunctions into more advanced queries) and how, under the hood, almost all PostgreSQL aggregates do a version of two-step aggregation. You’ll learn how the PostgreSQL implementation inspired us as we built TimescaleDB hyperfunctions, continuous aggregates, and other advanced features – and what this means for developers.If you’d like to get started with hyperfunctions right away,create your free trial accountand start analyzing 🔥. (TimescaleDB hyperfunctions are pre-installed on every Timescale Cloud instance, our hosted cloud-native relational time-series data platform).A primer on PostgreSQL aggregation (through pictures)When I first started learning about PostgreSQL 5 or 6 years ago (I was an electrochemist, and dealing with lots of battery data, as mentioned inmy last post on time-weighted averages), I ran into some performance issues. I was trying to better understand what was going on inside the database in order to improve its performance – and that’s when I foundBruce Momjian’s talks onPostgreSQL Internals Through Pictures. Bruce is well known in the community for his insightful talks (and his penchant for bow ties), and his sessions were a revelation for me.They’ve served as a foundation for my understanding of how PostgreSQL works ever since. He explained things so clearly, and I’ve always learned best when I can visualize what’s going on, so the “through pictures” part really helped - and stuck with - me.So this next bit is my attempt to channel Bruce by explaining some PostgreSQL internals through pictures. Cinch up your bow ties and get ready for some learnin'The author pays homage to Bruce Momjian (and looks rather pleased with himself because he’s managed to tie a bow tie on the first try).PostgreSQL aggregates vs. functionsWe have written abouthow we use custom functions and aggregates to extend SQL, but we haven’t exactly explained the differencebetweenthem.The fundamental difference between an aggregate function and a “regular” function in SQL is that anaggregateproduces a single result from agroupof related rows, while a regularfunctionproduces a result foreachrow:In SQL, aggregates produce a result from multiple rows, while functions produce a result per row.This is not to say that a function can’t have inputs from multiple columns; they just have to come from the same row.Another way to think about it is that functions often act on rows, whereas aggregates act on columns. To illustrate this, let’s consider a theoretical table `foo` with two columns:CREATE TABLE foo(
	bar DOUBLE PRECISION,
	baz DOUBLE PRECISION);And just a few values, so we can easily see what’s going on:INSERT INTO foo(bar, baz) VALUES (1.0, 2.0), (2.0, 4.0), (3.0, 6.0);The functiongreatest()will produce the largest of the values in columnsbarandbazfor each row:SELECT greatest(bar, baz) FROM foo; 
 greatest 
----------
        2
        4
        6Whereas the aggregatemax()will produce the largest value from each column:SELECT max(bar) as bar_max, max(baz) as baz_max FROM foo;

 bar_max | baz_max 
---------+---------
       3 |       6Using the above data, here’s a picture of what happens when we aggregate something:Themax()aggregate gets the largest value from multiple rows.The aggregate takes inputs from multiple rows and produces a single result. That’s the main difference between it and a function, but how does it do that? Let’s look at what it’s doing under the hood.Aggregate internals: row-by-rowUnder the hood, aggregates in PostgreSQL work row-by-row. But, then how does an aggregate know anything about the previous rows?Well, an aggregate stores some state about the rows it has previously seen, and as the database sees new rows, it updates that internal state.For themax()aggregate we’ve been discussing, the internal state is simply the largest value we’ve collected so far.Let’s take this step-by-step.When we start, our internal state isNULLbecause we haven’t seen any rows yet:Then, we get our first row in:Since our state isNULL, we initialize it to the first value we see:Now, we get our second row:And we see that the value of bar (2.0) is greater than our current state (1.0), so we update the state:Then, the next row comes into the aggregate:We compare it to our current state, take the greatest value, and update our state:Finally, we don’t have any more rows to process, so we output our result:So, to summarize, each row comes in, gets compared to our current state, and then the state gets updated to reflect the new greatest value. Then the next row comes in, and we repeat the process until we’ve processed all our rows and output the result.The max aggregate aggregation process, told in GIFs.There’s a name for the function that processes each row and updates the internal state: thestate transition function(or just “transition function” for short.) The transition function for an aggregate takes the current state and the value from the incoming row as arguments and produces a new state.It’s defined like this, wherecurrent_valuerepresents values from the incoming row,current_staterepresents the current aggregate state built up over the previous rows (or NULL if we haven’t yet gotten any), andnext_staterepresents the output after analyzing the incoming row:next_state = transition_func(current_state, current_value)Aggregate internals: composite stateSo, themax()aggregate has a straightforward state that contains just one value (the largest we’ve seen). But not all aggregates in PostgreSQL have such a simple state.Let’s consider the aggregate for average (avg):SELECT avg(bar) FROM foo;To refresh, an average is defined as:\begin{equation} avg(x) = \frac{sum(x)}{count(x)}  \end{equation}To calculate it, we store the sum and the count as our internal state and update our state as we process rows:The `avg()` aggregation process, told in GIFs. For `avg()`, the transition function must update a more complex state since the sum and count are stored separately at each aggregation step.But, when we’re ready to output our result foravg, we need to dividesumbycount:For some aggregates, we can output the state directly – but for others, we need to perform an operation on the state before calculating our final result.There’s another function inside the aggregate that performs this calculation: thefinal function. Once we’ve processed all the rows, the final function takes the state and does whatever it needs to produce the result.It’s defined like this, wherefinal_staterepresents the output of the transition function after it has processed all the rows:result = final_func(final_state)And, through pictures:How the average aggregate works, told in GIFs. Here, we’re highlighting the role of the final function.To summarize: as an aggregate scans over rows, itstransition functionupdates its internal state. Once the aggregate has scanned all of the rows, itsfinal functionproduces a result, which is returned to the user.Improving the performance of aggregate functionsOne interesting thing to note here: the transition function is called many, many more times than the final function: once for each row, whereas the final function is called once pergroupof rows.Now, the transition function isn’t inherently more expensive than the final function on a per-call basis – but because there are usually orders of magnitude more rows going into the aggregate than coming out, the transition function step becomes the most expensive part very quickly. This is especially true when you have high volume time-series data being ingested at high rates; optimizing aggregate transition function calls is important for improving performance.Luckily, PostgreSQL already has ways to optimize aggregates.Parallelization and the combine functionBecause the transition function is run on each row,some enterprising PostgreSQL developersasked:what if we parallelized the transition function calculation?Let’s revisit our definitions for transition functions and final functions:next_state = transition_func(current_state, current_value)

result = final_func(final_state)We can run this in parallel by instantiating multiple copies of the transition function and handing a subset of rows to each instance. Then, each parallel aggregate will run the transition function over the subset of rows it sees, producing multiple (partial) states, one for each parallel aggregate. But, since we need to aggregate over theentiredata set, we can’t run the final function on each parallel aggregate separately because they only have some of the rows.So, now we’ve ended up in a bit of a pickle: we have multiple partial aggregate states, and the final function is only meant to work on the single, final state - right before we output the result to the user.To solve this problem, we need a new type of function that takes two partial states and combines them into one so that the final function can do its work. This is (aptly) called thecombine function.We can run the combine function iteratively over all of the partial states that are created when we parallelize the aggregate.combined_state = combine_func(partial_state_1, partial_state_2)For instance, inavg, the combine function will add up the counts and sums.How parallel aggregation works, told in GIFs. Here, we’re highlighting the combine function (We’ve added a couple more rows to illustrate parallel aggregation.)Then, after we have the combined state from all of our parallel aggregates, we run the final function and get our result.DeduplicationParallelization and the combine function are one way to reduce the cost of calling an aggregate, but it’s not the only way.One other built-in PostgreSQL optimization that reduces an aggregate’s cost occurs in a statement like this:SELECT avg(bar), avg(bar) / 2 AS half_avg FROM foo;PostgreSQL will optimize this statement to evaluate the `avg(bar)` calculation only once and then use that result twice.And, if we have different aggregates with the same transition function but different final functions? PostgreSQL further optimizes by calling the transition function (the expensive part) on all the rows and then doing both final functions! Pretty neat!Now, that’s not all that PostgreSQL aggregates can do, but it’s a pretty good tour, and it’s enough to get us where we need to go today.Two-step aggregation in TimescaleDB hyperfunctionsIn TimescaleDB, we’ve implemented the two-step aggregation design pattern for our aggregate functions. This generalizes the PostgreSQL internal aggregation API and exposes it to the user via our aggregates, accessors, and rollup functions. (In other words, each of the internal PostgreSQL functions has an equivalent function in TimescaleDB hyperfunctions.)As a refresher, when we talk about the two-step aggregation design pattern, we mean the following convention, where we have an inner aggregate call:And an outer accessor call:The inner aggregate call returns the internal state, just like the transition function does in PostgreSQL aggregates.The outer accessor call takes the internal state and returns a result to the user, just like the final function does in PostgreSQL.We also have specialrollupfunctionsdefined for each of our aggregatesthat work much like PostgreSQL combine functions.PostgreSQL internal aggregation APIs and their TimescaleDB hyperfunctions’ equivalentWhy we use the two-step aggregate design patternThere are four basic reasons we expose the two-step aggregate design pattern to users rather than leave it as an internal structure:Allow multi-parameter aggregates to re-use state, making them more efficientCleanly distinguish between parameters that affect aggregates vs. accessors, making performance implications easier to understand and predictEnable easy to understand rollups, with logically consistent results, in continuous aggregates and window functions (one of our most common requests on continuous aggregates)Allow easierretrospective analysisof downsampled data in continuous aggregates as requirements change, but the data is already goneThat’s a little theoretical, so let’s dive in and explain each one.Re-using statePostgreSQL is very good at optimizing statements (as we saw earlier in this post, through pictures 🙌), but you have to give it things in a way it can understand.For instance,when we talked about deduplication, we saw that PostgreSQL could “figure out” when a statement occurs more than once in a query (i.e.,avg(bar)) and only run the statement a single time to avoid redundant work:SELECT avg(bar), avg(bar) / 2 AS half_avg FROM foo;This works because the `avg(bar)` occurs multiple times without variation.However, if I write the equation in a slightly different way and move the divisioninsidethe parentheses so that the expressionavg(bar)doesn’t repeat so neatly, PostgreSQLcan’tfigure out how to optimize it:SELECT avg(bar), avg(bar / 2) AS half_avg FROM foo;It doesn’t know that the division is commutative, or that those two queries are equivalent.This is a complicated problem for database developers to solve, and thus, as a PostgreSQL user, you need to make sure to write your query in a way that the database can understand.Performance problems caused by equivalent statements that the database doesn’t understand are equal (or that are equal in the specific case you wrote, but not in the general case) can be some of the trickiest SQL optimizations to figure out as a user.Therefore,when we design our APIs, we try to make it hard for users to unintentionally write low-performance code: in other words, the default option should be the high-performance option.For the next bit, it’ll be useful to have a simple table defined as:CREATE TABLE foo(
	ts timestamptz, 
	val DOUBLE PRECISION);Let’s look at an example of how we use two-step aggregation in thepercentile approximation hyperfunctionto allow PostgreSQL to optimize performance.SELECT 
    approx_percentile(0.1, percentile_agg(val)) as p10, 
    approx_percentile(0.5, percentile_agg(val)) as p50, 
    approx_percentile(0.9, percentile_agg(val)) as p90 
FROM foo;...is treated as the same as:SELECT 
    approx_percentile(0.1, pct_agg) as p10, 
    approx_percentile(0.5, pct_agg) as p50, 
    approx_percentile(0.9, pct_agg) as p90 
FROM 
(SELECT percentile_agg(val) as pct_agg FROM foo) pct;This calling convention allows us to use identical aggregates so that, under the hood, PostgreSQL can deduplicate calls to the identical aggregates (and is faster as a result).Now, let’s compare this to the one-step aggregate approach.PostgreSQL can’t deduplicate aggregate calls here because the extra parameter in theapprox_percentileaggregate changes with each call:So, even though all of those functions could use the same approximation built up over all the rows, PostgreSQL has no way of knowing that. The two-step aggregation approach enables us to structure our calls so that PostgreSQL can optimize our code, and it enables developers to understand when things will be more expensive and when they won't. Multiple different aggregates with different inputs will be expensive, whereas multiple accessors to the same aggregate will be much less expensive.Cleanly distinguishing between aggregate/accessor parametersWe also chose the two-step aggregate approach because some of our aggregates can take multiple parameters or options themselves, and their accessors can also take options:SELECT
    approx_percentile(0.5, uddsketch(1000, 0.001, val)) as median,--1000 buckets, 0.001 target err
    approx_percentile(0.9, uddsketch(1000, 0.001, val)) as p90, 
    approx_percentile(0.5, uddsketch(100, 0.01, val)) as less_accurate_median -- modify the terms for the aggregate get a new approximation
FROM foo;That’s an example ofuddsketch, anadvanced aggregation methodfor percentile approximation that can take its own parameters.Imagine if the parameters were jumbled together in one aggregate:-- NB: THIS IS AN EXAMPLE OF AN API WE DECIDED NOT TO USE, IT DOES NOT WORK
SELECT
    approx_percentile(0.5, 1000, 0.001, val) as median
FROM foo;It’d be pretty difficult to understand which argument is related to which part of the functionality.Conversely, the two-step approach separates the arguments to the accessor vs. aggregate very cleanly, where the aggregate function is defined in parenthesis within the inputs of our final function:SELECT
    approx_percentile(0.5, uddsketch(1000, 0.001, val)) as median
FROM foo;By making it clear which is which, users can know that if they change the inputs to the aggregate, they will get more (costly) aggregate nodes, =while inputs to the accessor are cheaper to change.So, those are the first two reasons we expose the API - and what it allows developers to do as a result. The last two reasons involve continuous aggregates and how they relate to hyperfunctions, so first, a quick refresher on what they are.Two-step aggregation + continuous aggregates in TimescaleDBTimescaleDB includes a feature calledcontinuous aggregates, which are designed to make queries on very large datasets run faster. TimescaleDB continuous aggregates continuously and incrementally store the results of an aggregation query in the background, so when you run the query, only the data that has changed needs to be computed, not the entire dataset.In our discussion of the combine functionabove,we covered how you could take the expensive work of computing the transition function over every row and split the rows over multiple parallel aggregates to speed up the calculation.TimescaleDB continuous aggregates do something similar, except they spread the computation work overtimerather than between parallel processes running simultaneously. The continuous aggregate computes the transition function over a subset of rows inserted some time in the past, stores the result, and then, at query time, we only need to compute over the raw data for a small section of recent time that we haven’t yet calculated.When we designed TimescaleDB hyperfunctions, we wanted them to work well within continuous aggregates and even open new possibilities for users.Let’s say I create a continuous aggregate from the simple table above to compute the sum, average, and percentile (the latter using a hyperfunction) in 15-minute increments:CREATE MATERIALIZED VIEW foo_15_min_agg
WITH (timescaledb.continuous)
AS SELECT id,
    time_bucket('15 min'::interval, ts) as bucket,
    sum(val),
    avg(val),
    percentile_agg(val)
FROM foo
GROUP BY id, time_bucket('15 min'::interval, ts);And then what if I come back and I want to re-aggregate it to hours or days, rather than 15-minute buckets – or need to aggregate my data across all ids? Which aggregates can I do that for, and which can’t I?Logically consistent rollupsOne of the problems we wanted to solve with two-step aggregation was how to convey to the user when it is “okay” to re-aggregate and when it’s not. (By “okay,” I mean you would get the same result from the re-aggregated data as you would running the aggregate on the raw data directly.)For instance:SELECT sum(val) FROM tab;
-- is equivalent to:
SELECT sum(sum) 
FROM 
    (SELECT id, sum(val) 
    FROM tab
    GROUP BY id) s;But:SELECT avg(val) FROM tab;
-- is NOT equivalent to:
SELECT avg(avg) 
FROM 
    (SELECT id, avg(val) 
    FROM tab
    GROUP BY id) s;Why is re-aggregation okay forsumbut not foravg?Technically, it’s logically consistent to re-aggregate when:The aggregate returns the internal aggregate state. The internal aggregate state for sum is(sum), whereas for average, it is(sum, count).The aggregate’s combine and transition functions are equivalent. Forsum(), the states and the operations are the same. Forcount(), thestatesare the same, but the transition and combine functionsperform different operationson them.sum()’s transition function adds the incoming value to the state, and its combine function adds two states together, or a sum of sums.  Conversely,count()s transition function increments the state for each incoming value, but its combine function adds two states together, or a sum of counts.But, you have to have in-depth (and sometimes rather arcane) knowledge about each aggregate’s internals to know which ones meet the above criteria – and therefore, which ones you can re-aggregate.With the two-step aggregate approach, we can convey when it is logically consistent to re-aggregate by exposing our equivalent of the combine function when the aggregate allows it.We call that functionrollup().Rollup()takes multiple inputs from the aggregate and combines them into a single value.All of our aggregates that can be combined haverollupfunctions that will combine the output of the aggregate from two different groups of rows. (Technically,rollup()is an aggregate function because it acts on multiple rows. For clarity, I’ll call them rollup functions to distinguish them from the base aggregate).  Then you can call the accessor on the combined output!So using that continuous aggregate we created to get a 1-day re-aggregation of ourpercentile_aggbecomes as simple as:SELECT id, 
    time_bucket('1 day'::interval, bucket) as bucket, 
    approx_percentile(0.5, rollup(percentile_agg)) as median
FROM foo_15_min_agg
GROUP BY id, time_bucket('1 day'::interval, bucket);(We actually suggest that you create your continuous aggregates without calling the accessor function for this very reason. Then, you can just create views over top or put the accessor call in your query).This brings us to our final reason.Retrospective analysis using continuous aggregatesWhen we create a continuous aggregate, we’re defining a view of our data that we then could be stuck with for a very long time.For example, we might have a data retention policy that deletes the underlying data after X time period. If we want to go back and re-calculate anything, it can be challenging, if not impossible, since we’ve “dropped” the data.But, we understand that in the real world, you don’t always know what you’re going to need to analyze ahead of time.Thus, we designed hyperfunctions to use the two-step aggregate approach, so they would better integrate with continuous aggregates. As a result, users store the aggregate state in the continuous aggregate view and modify accessor functionswithoutrequiring them to recalculate old states that might be difficult (or impossible) to reconstruct (because the data is archived, deleted, etc.).The two-step aggregation design also allows for much greater flexibility with continuous aggregates. For instance, let’s take a continuous aggregate where we do the aggregate part of the two-step aggregation like this:CREATE MATERIALIZED VIEW foo_15_min_agg
WITH (timescaledb.continuous)
AS SELECT id,
    time_bucket('15 min'::interval, ts) as bucket,
    percentile_agg(val)
FROM foo
GROUP BY id, time_bucket('15 min'::interval, ts);When we first create the aggregate, we might only want to get the median:SELECT
    approx_percentile(0.5, percentile_agg) as median
FROM foo_15_min_agg;But then, later, we decide we want to know the 95th percentile as well.Luckily, we don’t have to modify the continuous aggregate; wejust modify the parameters to the accessor function in our original query to return the data we want from the aggregate state:SELECT
    approx_percentile(0.5, percentile_agg) as median,
    approx_percentile(0.95, percentile_agg) as p95
FROM foo_15_min_agg;And then, if a year later, we want the 99th percentile as well, we can do that too:SELECT
    approx_percentile(0.5, percentile_agg) as median,
    approx_percentile(0.95, percentile_agg) as p95,
    approx_percentile(0.99, percentile_agg) as p99
FROM foo_15_min_agg;That’s just scratching the surface. Ultimately, our goal is to provide a high level of developer productivity that enhances other PostgreSQL and TimescaleDB features, like aggregate deduplication and continuous aggregates.An example of how the two-step aggregate design impacts hyperfunctions’ codeTo illustrate how the two-step aggregate design pattern impacts how we think about and code hyperfunctions, let’s look at thetime-weighted average family of functions. (Ourwhat time-weighted averages are and why you should carepost provides a lot of context for this next bit, so if you haven’t read it, we recommend doing so. You can also skip this next bit for now.)The equation for the time-weighted average is as follows:\begin{equation}  time\_weighted\_average = \frac{area\_under\_curve}{ \Delta T}   \end{equation}As we noted in thetable above:time_weight()is TimescaleDB hyperfunctions’ aggregate and corresponds to the transition function in PostgreSQL’s internal API.average()is the accessor, which corresponds to the PostgreSQL final function.rollup()for re-aggregation corresponds to the PostgreSQL combine function.Thetime_weight()function returns an aggregate type that has to be usable by the other functions in the family.In this case, we decided on aTimeWeightSummarytype that is defined like so (in pseudocode):TimeWeightSummary = (w_sum, first_pt, last_pt)w_sumis the weighted sum (another name for the area under the curve), andfirst_ptandlast_ptare the first and last (time, value) pairs in the rows that feed into thetime_weight()aggregate.Here’s a graphic depiction of those elements, which builds on ourhow to derive a time-weighted average theoretical description:Depiction of the values we store in the `TimeWeightSummary` representation.So, thetime_weight()aggregate does all of the calculations as it receives each of the points in our graph and builds a weighted sum for the time period (ΔT) between the first and last points it “sees.” It then outputs theTimeWeightSummary.Theaverage()accessor function performs simple calculations to return the time-weighted average from theTimeWeightSummary(in pseudocode wherept.time()returns the time from the point):func average(TimeWeightSummary tws) 
	-> float {
		delta_t = tws.last_pt.time - tws.first_pt.time;
		time_weighted_average = tws.w_sum / delta_t;
		return time_weighted_average;
	}But, as we built thetime_weighthyperfunction, ensuring therollup()function worked as expected was a little more difficult – and introduced constraints that impacted the design of ourTimeWeightSummarydata type.To understand the rollup function, let’s use our graphical example and imagine thetime_weight()function returns twoTimeWeightSummariesfrom different regions of time like so:What happens when we have multiple TimeWeightSummaries representing different regions of the graph.Therollup()function needs to take in and return the sameTimeWeightSummarydata type so that ouraverage()accessor can understand it. (This mirrors how PostgreSQL’s combine function takes in two states from the transition function and then returns a single state for the final function to process).We also want therollup()output to be the same as if we had computed thetime_weight()over all the underlying data. The output should be aTimeWeightSummaryrepresenting the full region.TheTimeWeightSummarywe output should also account for the area in the gap between these two weighted sum states:Mind the gap! (between oneTimeWeightSummaryand the next).The gap area is easy to get because we have the last1and first2points - and it’s the same as thew_sumwe’d get by running thetime_weight()aggregate on them.Thus, the overallrollup()function needs to do something like this (wherew_sum()extracts the weighted sum from theTimeWeightSummary):func rollup(TimeWeightSummary tws1, TimeWeightSummary tws2) 
	-> TimeWeightSummary {
		w_sum_gap = time_weight(tws1.last_pt, tws2.first_pt).w_sum;
		w_sum_total = w_sum_gap + tws1.w_sum + tws2.w_sum;
		return TimeWeightSummary(w_sum_total, tws1.first_pt, tws2.last_pt);
	}Graphically, that means we’d end up with a singleTimeWeightSummaryrepresenting the whole area:The combinedTimeWeightSummary.So that’s how the two-step aggregate design approach ends up affecting the real-world implementation of our time-weighted average hyperfunctions. The above explanations are a bit condensed, but they should give you a more concrete look at howtime_weight()aggregate,average()accessor, androllup()functions work.Summing it upNow that you’ve gotten a tour of the PostgreSQL aggregate API,  how it inspired us to make the TimescaleDB hyperfunctions two-step aggregate API, and a few examples of how this works in practice, we hope you'll try it out yourself and tell us what you think :).If you’d like to get started with hyperfunctions right away,spin up a fully managed TimescaleDB service and try it for free.Hyperfunctions are pre-loaded on each new database service on Timescale Cloud, so after you create a new service, you’re all set to use them!If you prefer to manage your own database instances, you candownload and install the timescaledb_toolkit extensionon GitHub, after which you’ll be able to use `time_weight` and all other hyperfunctions.If you have questions or comments on this blog post,we’ve started a discussion on our GitHub page, and we’d love to hear from you. (And, if you like what you see, GitHub ⭐ are always welcome and appreciated too!)We love building in public, and you can view ourupcoming roadmap on GitHubfor a list of proposed features, features we’re currently implementing, and features available to use today. For reference, the two-step aggregate approach isn’t just used in the stabilized hyperfunctions covered here; it’s also used in many of our experimental features, includingstats_agg()uses two-step aggregation to make simple statistical aggregates, like average and standard deviation, easier to work with in continuous aggregates and tosimplify computing rolling averages.counter_agg()uses two-step aggregation to make working with counters more efficient and composable.Hyperlogloguses two-step aggregation in conjunction with continuous aggregates to give users faster approximate COUNT DISTINCT rollups over longer periods of time.These features will be stabilized soon, but we’d love your feedback while the APIs are still evolving. What would make them more intuitive? Easier to use?Open an issueorstart a discussion!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-postgresql-aggregation-works-and-how-it-inspired-our-hyperfunctions-design-2/
2020-04-17T13:47:23.000Z,"Our Product Team's Reading List: What Moral Psychology, the Rust Belt, and Framing Can Teach Us About UX","Like many people, we've been catching up on our book backlog – and we're sharing our recent reads, plus what they've taught us about our work as product managers and engineers.In light of COVID-19 and everyone’s weekend plans put on pause, my fellow Timescale Product team members and I have been using this extra time to catch up on reading.In the event you’re anything like us and keen to spend this time checking items off your literary bucket list, we thought it would be fun to share what we’ve been reading lately, with one or two Product Management lessons we’ve learned from each title.Our book list isn’t necessarily Product-related, but we’re firm believers that a great way to excel in our day-to-day work comes from the ability to connect disparate ideas gathered from other facets of our lives. In our case, you’ll see a general nerdiness and tendency towards popular science – and we’re keen to get recommendations from you (we know our picks are a bit mainstream).Our list:Hillbilly Elegy (J.D. Vance)The Upside of Stress (Kelly McGonigal)The Righteous Mind (Jonathan Haidt)Factfulness (Hans Rosling)Loonshots (Safi Bahcall)Competing Against Luck (Clayton Christensen)Hillbilly Elegy (J.D. Vance)Hillbilly Elegyis a fantastic read, full of deeply personal stories of the anti-patterns that have prevented communities in the Rust Belt (an area of the US that’s been hard hit by the decline in industrial jobs) from bouncing back.As an example, the author mentions a concept called “Adverse Childhood Event” (ACE). ACEs are extremely traumatic events (like alcoholism, food insecurity, etc.) that strike during someone’s formative years. In the author’s words...ACEs happen everywhere, in every community. But studies have shown that ACEs are far more common in my corner of the demographic world.A report by the Wisconsin Children’s Trust Fund showed that among those with a college degree or more (the non–working class), fewer than half had experienced a single ACE. Among the working class, well over half had at least one ACE, while about 40 percent had multiple ACEs. This is shocking.To us, this emphasized the path-dependency inherent in so many people’s current circumstances and just how critical empathy is to understanding them. While it’s a nuanced concept not directly translating into product management, it more so reminds us that there is always a human being, with unique experiences and perspectives, on the other side of the screen.Note: You can take the ACE quizhereif you’re interested in learning more.The Upside of Stress (Kelly McGonigal)In short, this book makes the case that stress is actually a positive thing in our lives. It is a “feature” of our evolution - not a bug - and simply viewing it as such is all it takes to capture the benefits.McGonigal includes numerous research studies and examples, but two case studies in particular illustrate the impact of framing and positioning (and thus have interesting UX implications):Compared to a control group, housekeepers at a large hotel chain lost weight when much of their daily work was reframed to actually be exercise (e.g., lifting furniture, taking thousands of walking steps, etc.). This was only a framing change: actions they previously viewed as their “daily labor,” were now viewed completely differently, as a way to positively impact their health.Study participants received identical smoothies with different non-brand names, and the smoothie with “Indulgent” in its name stimulated a meaningful larger amount of a stress and hunger hormone, ghrelin. In other words, participants were willing to consume a different number of calories, based on name change alone.The Righteous Mind (Jonathan Haidt)One of our teammates loves to ask personal heroes “which book permanently changed the way you think about the world?” In the event we’re ever asked,The Righteous Mindwould certainly make our list.The book touches on a broad range of topics, describing the psychology of morality and moral decision-making. One interesting takeaway, similar to Kahneman's concept ofSystem I and System II thinking, is the author’s description of moral decision-making as one being made “by an elephant and its rider.”Haidt explains that people always make an initial, intuitive judgement on moral issues (the elephant) which is then only justified and explained (by the rider).But, logical reasoning is almost never involved as the prime mover.Haidt also likens the rider to our own personal press secretary, who searches for valid reasons why we’ve done or think somethingafter the fact.One entertaining study illustrated “moral dumbfounding,” or scenarios where the rider (our rational self) can’t easily find a way to justify the elephant’s (our moral self) reactions.Haidt asked subjects whether they would drink a hypothetical cup of juice which had a completely clean, sterilized, lab-grown cockroach dipped into it.A large portion of interviewees still said “no,” even after the experimenters removed every possible excuse (e.g., the cockroach is sterilized and thus safe).Yet, interviewees just couldn’t explain why they still wouldn’t drink it.Moral thinking seems to trigger in more scenarios than one would expect, which is where we saw the potential connection to product management. For example:Haidt's moral category of “Fairness” ties into pricing and how we balance perceived value with monetary cost.The moral of “Authority” is loosely related to how flagship users motivate more people to sign up or use your product.The moral of  “Loyalty” - both your users’ and within your own company or team - plays a role in how you position your product externallyandhow you make decisions and tradeoffs as you build new features.If you want to explore more about Moral Foundations and see where you fall on various scales, Haidt’sYourMorals.Orgis still up and running.Factfulness (Hans Rosling)Factfulnesscovers many topics, but the core argument is that the world is statistically better than most of us think it is. From there, Rosling discusses how and why to discard preconceived notions and let facts and data guide you.As a PM, this approach is invaluable and central to our work: we need to unite stakeholders on at least a few, data-driven ground truths to drive group decision-making. While a persuasive approach that also uses Pathos in some way is more effective than just raw data, that first alignment step is crucial.Note: If you’re interested in diving into an academic rabbit hole to see if the world isactuallygetting better, we recommend exploring an ongoing debate between Nassim Taleb and Steven Pinker who take opposite positions. A brief summary of the debate is foundhere.Loonshots (Safi Bahcall)In short,Loonshotsexplains and characterizes a more realistic view on how to nurture “moonshots” and the associated pushback they often receive. In the author’s words:“Everybody knows what a moonshot is. It’s a big idea, a destination, or a goal like curing cancer, eliminating poverty or something. But the question is, how do we get there? It turns out that if you look back in history at the big ideas,the ones that have changed the course of science, business, or history, they rarely arrived with blaring trumpets and red carpets dazzling everybody with their brilliance.They’re usually dismissed,sometimes for years, sometimes for decades with their champions written off as crazy. Since there wasn’t any better word in the English language, I just made one up and I called them loonshots.”We saw a lot of valuable ideas inLoonshotsfrom our past experiences working in larger companies. Being able to make recommendations to the business around how to foster innovation and how to spot pockets of innovation (before leadership even knows it's there) is extremely valuable as a PM, as is being able to recognize and find like-minded individuals.Even if you’re at a small startup like us, learning a few tactics to address pushback when you’re trying to change the status quo (and why people might push back in the first place)? Infinitely valuable.Competing Against Luck (Clayton Christensen)Competing Against Luckis the book that codified and popularized the “jobs-to-be-done” framework for product management. In essence, it is helpful to think of your customers as “hiring” your product to solve a specific job for them.This was less of an extra-curricular book - it’s directly tied to our PM work, so our list of takeaways isverylong. A few memorable anecdotes:A fast food chain hoping to sell more milkshakes was looking to discover the key product dimensions they should change. Maybe add name-brand candy? Different sizes? In reality, they discovereda valuable “job” their milkshakes were being hired to solve was for consumers who wanted something fast, filling, and hands-free to consume on their drive to work. This realization had sweeping implications for the product and competitor set. For example, the smoothie is actually “competing” with a breakfast burrito, so we should add elements that make it seem like the better choice in that context.The often-cited example of “customers don’t buy a quarter inch drill, they buy a quarter inch hole.”When Procter and Gamble (Pampers) started selling diapers in China, their initial differentiator was intended to be “cheaper price.” There was a broader problem though - the disposable diaper market didn’t exist in China. It was culturally more accepted to use cloth diapers. After a couple missteps, Pampers’ messaging shifted to focus heavily on the diaper’s ability to help babies sleep better (a more compelling “job to be done” for the user with the exact same product). Pampers’ team used studies to demonstrate that babies who slept better became smarter and consequently outperformed in school – and this shift helped fuel a multi-year, meteoric rise in Pampers’ adoption in China.Note:Intercomhas a wealth of content on the jobs-to-be-done framework which is used throughout the company.Honorable mentionsCheck outthis threadon Ajay Kulkarni’s Twitter for other recommendations from our community.Wrapping UpIf you’ve read any of the above and have additional comments, suggestions for new books to add to our shelves, or just want to discuss all things product management,reach out on Slack(I’m@Alex Thinath, and the whole team is active on all channels).To learn what we’re working on when we’re not catching up on our reading, you can see our latest release notes and database roadmaphere(andsubscribe for updatesto get notified about new releases, beta programs, and beyond).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/our-product-teams-reading-list-what-moral-psychology-the-rust-belt-and-framing-can-teach-us-about-ux/
2022-06-23T13:03:01.000Z,Slow Grafana Performance? Learn How to Fix It Using Downsampling,"Downsampling in GrafanaGraphs are awesome. They allow us to understand data quicker and easier, highlighting trends that otherwise wouldn’t stand out. And Grafana, the open-source visualization tool, is a fantastic tool for creating graphs, especially for time-series data.If you have some data that you want to analyze visually, you just hook it up to your Grafana instance, set up your query, and you’re off to the races. (If you’re new to Grafana and Timescale, don’t worry, we’ve got you covered. See our Getting Started with Grafana and TimescaleDBdocsorvideosto get up and running).However, while Grafana is an awesome tool for generating graphs, problems still arise when we have too much data. Extremely large datasets can be prohibitively slow to load, leading to frustrated users or, worse, unusable dashboards.These largetime-series datasetsare especially common in industries like financial services, the Internet of Things, andobservabilityas data can be relentless, often generated at high rates and volumes.To better understand the problems that can occur when we have extremely large datasets, consider the example of stock ticker data and this graph showing 30 days' worth of trades for five different stocks (AAPL, TSLA, NVDA, MSFT, and AMD):This graph is composed of five queries which collectively contain nearly 1.3 million data points and takes nearly 20 seconds to load, pan, or zoom!Even with more manageable amounts of data, our graphs can still sometimes be difficult to interpret if the data is too noisy. If the daily variance of our data is so high, it can hide the underlying trends that we're looking for. Consider this graph showing the volume of taxi trips taken in New York City over a two-month period:That spike a third of the way in may be a significant shift in volume, and those lower peaks toward the right edge might be a significant decline. It's not immediately obvious though, and certainly, this is not the powerful tool we want our graphs to be.We can use different types of downsampling to solve the problems of slow-loading Grafana dashboards and noisy graphs, respectively. Downsampling is the practice of replacing a large set of data points with a smaller set.We’ll implement our solutions using two ofTimescaleDB’s hyperfunctionsfor downsampling, making it easy to manipulate and analyze time-series data with fewer lines of SQL code. We’ll look at one hyperfunction for downsampling using the Largest Triangle Three Buckets orlttb()method, and another for downsampling using the ASAP smoothing algorithm, both of which come pre-installed with Timescale or can be accessed via thetimescaledb_toolkit extensionif you self-manage your database.Example 1: Load faster dashboards with lttb( ) downsamplingIn our first example, which plots the prices for five stocks over a 30-day period, the problem is that we have way too much data, resulting in a slow-loading graph. This is because thereal-time stocks datasetwe’re using has upwards of 10,000 points per day for each stock symbol!Given the timeframe of our analysis (30 days), this is far more data than we need to spot a trend, and the time needed to load this graph is dominated by the cost of fetching all of the data.To solve this problem, we need to find a way to reduce the number of data points we're getting from our data source. Unfortunately, doing this in a manner that doesn't drastically deform our graph is actually a very tricky problem. For example, let’s look at just the NVDA ticker price:Here's what we see if we just naively take the 10-minute average for the NVDA symbol (overlaid in yellow on the original data).The graph of the average (mean) roughly follows the underlying data but completely smooths away almost all of the peaks and valleys, and those are the most interesting parts of the dataset! Taking the first or last point from each bucket results in an even more skewed graph, as the outlying points have no weight unless they happen to fall in just the right spot.What we need is a way to capture the most interesting point from each bucket. To do that, we can use thelttb()algorithmwhich gives us a downsampled graph that follows the pattern of the original graph quite closely. (As an aside,lttb()was invented bySveinn Steinarssonin his master’s thesis).Usinglttb(), the downsampled data is barely distinguishable from the original,despite having less than 0.5 % of the points!lttb()works by keeping the same first and last point as the original data but dividing the rest of the data into equal intervals. For each interval, it then tries to find the most impactful point. It does this by building a triangle for each point in the interval with the point selected from the previous interval and the average of the points in the next interval. These triangles are compared with one another by area. The largest resulting triangle corresponds to the point in the interval that has the largest impact on how the graph looks.As we see above, the result is a graph that very closely resembles the original graph. What's not as obvious is that the raw data was nearly 315,000 rows of data that took over five seconds to pull into our dashboard.  Thelttb()data was 1,404 rows that took less than one second to fetch.Here is the SQL query we used in our Grafana panel to get thelttb()data.SELECT
  time AS ""time"",
  value AS ""NVDA lttb""
FROM unnest((
    SELECT lttb(time, price, 2 * (($__to - $__from) / $__interval_ms)::int)
    FROM stocks_real_time
    WHERE symbol = 'NVDA' AND $__timeFilter(""time""))
)
  ORDER BY 1;As you can see, the real work here is done by thelttb()hyperfunctioncall in the innerSELECT.  This function takes thetimeandvaluecolumns from our table, and also a third integer specifying the target resolution, which is the number of points it should return.Unfortunately, Grafana doesn't directly expose the panel width in pixels to us, but we can get an approximation from the$__intervalglobal variable(which is approximately(to - from) / resolution). For this graph, the interval was a bit of an underestimation, hence us doubling it in the function above.Ourlttb()hyperfunctionreturns a customtimevectorobject, which usesunnestto gettime,valuerows that Grafana can understand and plot.Example 2: Find signal from noisy datasets with ASAP smoothing downsamplinglttb()is a fantastic downsampling algorithm for giving us a subset of points that maintain the visual appearance of a graph. However, sometimes the problem is that the original graph is so noisy that the long-term trends we're trying to see are lost in the normal periodic variance of the data. This is the case we saw in our second example above, that of taxi data (and shown below):In this case, what we're interested in isn't a way of just reducing the number of points in a graph (as we saw before, that ends up with a graph that looks the same!), but doing so in a manner that smooths away the noise.We can use a downsampling technique calledAutomated Smoothing for Attention Prioritization (ASAP), which was developed byKexin RongandPeter Bailis.ASAP works by analyzing the data for intervals of high autocorrelation.Think of this as finding the size of the repeating shape of a graph, so maybe 24 hours for our taxi data, or even 168 hours (one week). Once ASAP has found the range with the highest autocorrelation, it will smooth out the data by computing a rolling average using that range as the window size.For instance, if you have perfectly regular data, ASAP should mostly smooth everything away to the underlying flat trend, as in the following example:The green line here is the raw data. It is generated as a sine wave with an interval of 20 and an offset of 100 that repeats daily. The yellow line is the ASAP algorithm applied to the data, showing that the graph is entirely regular noise with no interesting underlying fluctuation.Obviously ASAP can work well on this type of synthetic data, but let's see how it does with our taxi data.Here it becomes very obvious that there was a significant dip over from about 11/26 to 12/03, which happens to be Thanksgiving weekend, a US public holiday weekend that occurs at the end of November every year. We can see this even more dramatically by selecting only the ASAP output and letting Grafana auto-adjust the scale:The data for this graph is thetaxi trips CSV file. The SQL query we're running in Grafana is this:SELECT
  time AS ""time"",
  value AS ""asap""
FROM unnest((
  SELECT asap_smooth(time, value, (($__to - $__from) / '$__interval_ms')::integer)
  FROM taxidata
  WHERE $__timeFilter(""time""))
)
ORDER BY 1As in example 1 above, theasap_smoothhyperfunction does most of the work here, taking the time and value columns, as well as a target resolution as arguments. We use the same trick from example 1 to approximate the panel width from Grafana's global variables.Learn MoreEager to try downsampling or learn more about other hyperfunctions? Check out ourdownsampleandhyperfunctionsdocs for more information on how hyperfunctions can help you efficiently query and analyze your data.Looking for more Grafana guides? Here are ourGrafana tutorialsand ourGrafana 101 Creating Awesome Visualizationsfor more support on visualizations in Grafana.If you need a database to store your time-series data and power your dashboards, tryTimescale, our fast, easy-to-use, and reliable cloud-native data platform for time series built on PostgreSQL. (You can sign up for a 30-day free trial, no credit card required.)Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/slow-grafana-performance-learn-how-to-fix-it-using-downsampling/
2020-08-06T14:18:14.000Z,How to visualize timeshifts to compare metrics over time in Grafana using PostgreSQL,"Learn how (and why) to combine PostgreSQL, TimescaleDB, and Grafana to visualize timeshifts and compare how your metrics change over time.The problem: Comparing metrics over time (aka timeshifting)When we’re doing real-time monitoring or historical analysis, we often want to visually compare the value of a metric NOW to the value X days, weeks, hours, or months ago (or, in other words, we want to compare its value at the current time to its valuetimeshiftedone or more intervals of time ago).This is known as atimeshift:comparing a metric against itself, but for a different time period.This is especially common in DevOps, IoT, and user behavior analysis scenarios, where we want to understand if things like upticks or downticks are seasonal, or a result of something new – as well as a host of other questions that require us to analyze how certain metrics change over time.For example, take the case of monitoring taxi rides. On any given day, we might ask things like: how does the ride activity today compare with activity over the last 3 days? Or, how does ride activity this Friday compare to Friday last week?  What about the week before? Or the same time last year? These questions about taxi rides could easily apply to our website uptime metrics, our CPU utilization, and so forth...One (painful) way to answer these questions might be to create separate graphs for each time interval and manually compare them by eye. However, this isn’t very efficient, and manual comparison can be mentally taxing.Graphs comparing taxi rides taken in week 1 and week 2 of January 2016. Notice how difficult it is to compare ride activity between the two graphs.The solution: Use PostgreSQL LATERAL JOINA better way would be to have all trend lines (both for current activity and timeshifted activity) on a single graph. However, inGrafana, this isn't always possible, depending on which datasource you use. For example, Grafana’sGraphite datasource supports timeshift natively, but many others do not.For the PostgreSQL datasource, timeshifting is possible, and the best way to create time-shifted graphs is to use PostgreSQL’sLATERAL JOINfunction.Using theLATERAL JOINfunction, we can create timeshifted graphs for monitoring and historical analysis like these:Timeshifted graph showing taxi rides for today (green) and last 3 days.Timeshifted graph showing taxi rides for given day (yellow line) and previous week (green line)Try It Yourself: Implementation in Grafana & Sample QueriesTo help you get the hang of creating timeshifted graphs on a sample dataset before applying it to your own projects, I’ve put together this handy step-by-step guide.ScenarioWe’ll use the use case of monitoring IoT devices, specifically taxis equipped with location-detecting sensors. Our dataset comes from theNew York Taxi and Limousine Commission(NYC TLC) for the month of January 2016.PrerequisitesTimescaleDB instance(Managed Service for TimescaleDB or self-hosted), running PostgreSQL 11+Grafana instance(cloud or self-hosted)TimescaleDB instance connected to Grafana (seeour Grafana setup tutorial)To load the taxi dataset into TimescaleDB, completeMission 1 in this tutorial, which will take you through downloading the .CSV file and inserting the data into your database.Example 1: Building a 3 Day TimeshiftLet’s say we wanted to answer: “how does taxi ride activity today compare with the activity from the previous 3 days?”Here’s the full query, with annotations, showing how to use the PostgreSQLLATERAL JOINfunction to create a graph that displays thecurrent numberof rides, as well astimeshifted rides from the previous 3 days.-- What to name the series
SELECT time, ride_count, CASE WHEN step = 0 THEN 'today' ELSE (-interval)::text END AS metric
FROM
-- sub-query to generate the intervals
    ( SELECT step, (step||'day')::interval AS interval FROM generate_series(0,3) g(step)) g_offsets
    JOIN LATERAL (
-- subquery to select the rides 
    SELECT
-- adding set interval to time values
      time_bucket('15m',pickup_datetime + interval)::timestamptz AS time, count(*) AS ride_count FROM rides
-- subtract value of interval from time to plot
-- today = 0, 1 day ago = 1, etc
    WHERE pickup_datetime BETWEEN $__timeFrom()::timestamptz - interval AND $__timeTo()::timestamptz - interval
    GROUP BY 1
    ORDER BY 1
    ) l ON trueQuery to plot rides in 15 minute intervals, with timeshifts for the previous 3 daysThis produces the following graph:Graph showing taxi rides taken in January 2016, timeshifted to compare rides today to prior three days. Today’s rides shown in green, -1 day in red, -2 days in blue, -3 days in yellow.If we zoom into a 2 day time period (by selecting it using the timepicker or highlighting it in the graph), we can see how timeshifting allows us to compare ride activity, simply by hovering over the graph at any given time interval:Graph showing taxi rides taken in January 12 and January 13 2016, timeshifted to compare rides today to the prior three days, zoomed in to an arbitrary 2 day period. Today’s rides shown in green, -1 day in red, -2 days in blue, -3 days in yellow.How the query works:In this query, theLATERAL JOINfunctions like a “for each” loop, making the results of the sub-querybeforetheLATERAL JOINavailable to each result of the sub-query which comes after it.In this case, the querybeforetheLATERAL JOINgenerates the intervals we want to compare ride activity over. We generate the intervals of 0,1,2 and 3 days, since we want to compare ride behaviour on any given day, to that of the previous 3 days:-- sub-query to generate the intervals
( SELECT step, (step||'day')::interval AS interval 
FROM generate_series(0,3) g(step)) g_offsetsQuery to generate intervals to compare ride activity overIn the queryaftertheLATERAL JOIN, we plot the number of rides in our time period of interest in 15 minute time buckets. Notice how we use ourintervalvalue from the previous sub-query: by adding and subtracting theintervalin thetime_bucketfunction and in theWHEREclause to filter the time-range for the rides selected, we’re able to get the correct values for current and timeshifted intervals:-- subquery to select the rides 
(SELECT
-- adding set interval to time values  time_bucket('15m',pickup_datetime + interval)::timestamptz AS time, count(*) AS ride_count 
FROM rides
-- subtract value of interval from time to plot
-- today = 0, 1 day ago = 1, etc
WHERE pickup_datetime BETWEEN $__timeFrom()::timestamptz - interval 
AND $__timeTo()::timestamptz - interval
GROUP BY 1
ORDER BY 1)Query to we plot the number of rides in 15 minute time buckets for current and timeshifted periodsFor more onLATERAL JOIN, see this usefultutorialfrom the folks at Heap and the official PostgreSQLdocs.Visual Pro-tip: Add a series overrideIn order to make it easier to distinguish between rides for any given day and rides from the previous 3 days, we can apply a series override in order modify the appearance of the timeshifted lines:Series override parameters to distinguish between real and timeshifted linesUsing the parameters above, we apply a series override to the timeshifted series in order to give them a smaller line width than the real line, allowing us to distinguish between them more easily. We can then look of the non-timeshifted line under the Display settings -- in the image below line width is set to 5 and line are to 2:Final graph for current rides and previous 3 day timeshift with visual treatment appliedExample 2: Building a 1 Week TimeshiftsNext, we want to answer:“How does the activity this week compare to last week?”In this example, we create a graph to display the current number of rides, as well as a timeshifted line to graph the rides from the previous week.Much of the query is the same as in Example 1; the only differences are (1) the interval definition changes fromdaytoweekand (2) the series we generate only has two values, 0 and 1, since we only want to compare to the previous week (vs. the 3 day period in the prior example.)SELECT time, ride_count, 
	CASE WHEN step = 0 THEN 'today' ELSE (-interval)::text END AS metric
FROM
    ( SELECT step, (step||'week')::interval AS interval FROM generate_series(0,1) g(step)) g_offsets
JOIN LATERAL (
    SELECT
      time_bucket('15m',pickup_datetime + interval)::timestamptz AS time, count(*) AS ride_count FROM rides
    WHERE
      pickup_datetime BETWEEN $__timeFrom()::timestamptz - interval AND $__timeTo()::timestamptz - interval
    GROUP BY 1
    ORDER BY 1
) l ON trueQuery to plot current rides and 1 week timeshifted ridesThis produces the following graph:Graph showing taxi rides taken in January 2016, time-shifted to compare rides today to the previous week, zoomed in to an arbitrary 5 day period. Rides for a given day are shown in green and rides on that day last from the previous week are shown in yellowVisual Pro-tip: Add a series overrideTo make it easier to distinguish between rides for any given day and rides from the previous week, we can apply a series override that modifies the appearance of the timeshifted lines:Graph showing current rides and 1 week timeshifted rides with series override applied to visually distinguish betweeen timeshift and non-timeshifted linesTo achieve this, we set the line area to 0 (under Display settings), and then apply a series override to the timeshifted series.In the series override settings, we:Set the line fill to 2, giving us a shadow lookSet the line width to 0, leaving thenon-timeshifted graph as the only series with a solid line, making it more distinguishable.Series override settings for making time-shifted lines more distinguishableNext StepsIn this tutorial, we covered what timeshifting is, how it works, and how to use PostgreSQLLATERAL JOIN, TimescaleDB, and Grafana to visualize timeshifts to easily compare data across two (or more!) time periods.⏰  To modify the query to timeshift any arbitrary number of minutes, hours, days, months, or years, change the parameters ongenerate_seriesandintervaldefinition (while it’s most common to compare metrics NOW to previous periods, you can use time-shifting to compare ANY two time periods).Happy timeshifting!Learn MoreWant more Grafana tips? Explore ourGrafana tutorials(I recommend this one onvariablesand this one onvisualizing missing data).Need a database to power your dashboarding and data analysis? Get started withTimescale Cloud(it’s our fast, easy-to-use, and reliable cloud-native data platform for time-series, built on PostgreSQL,). When you sign up, you’ll get 30 days of completely free use to get you and up and running.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grafana-postgres-timeshift/
2020-04-13T14:38:48.000Z,Your Questions Answered (YQA): April Edition,"This month brought us a wide range of community questions, from optimizing queries for Grafana to effective indexing, partitioning data, and more. See our product & engineering team's responses – and ask your own at an upcoming session.In “Your Questions Answered,”  we share - and answer - some of the most interesting, generally applicable, or frequently asked questions from the latest Office Hours sessions, virtual events, and our Slack community.Have a question you’d like answered?Sign up for an upcoming Office Hours session(first Tuesday of each month @ 9am PT / 12pm ET / 4 pm GMT)Join our Slack channel(our engineers are active in all channels)Start a conversation on Twitter (@timescaleDB)Let's get to it!How do I optimize my queries for Grafana?We recommend optimizing your queries, using a SQL development tool likepgAdminorDBeaver, before you port anything to Grafana to ensure that queries perform as expected. Once you’re confident that you’re getting the results you expect and need, you can build dashboards accordingly.Additionally, TimescaleDB includes various capabilities to optimize your queries, including:continuous data aggregation,downsampling, andadvanced columnar compression. We'd also recommend checking outour step-by-step Continuous Aggregates tutorialas you get started.And, if you’re interested in learning more about creating Grafana dashboards for your time-series datasets, join us on April 22nd for “Guide to Grafana 101: Getting started with (awesome) visualizations.” We’ll walk through the UI, then demo when, why, and how to use various types of charts, gauges, histograms, and more.What’s the most effective indexing to use?Our experience has shown that for time-series data, the most-useful index type varies widely depending on your data.However, we typically encourage developers to start with our default indexes; TimescaleDB automatically creates a time index on your data when a hypertable is created.  Equally important, we outlinevarious best practices and considerations in our docs(including indexing columns with discrete values and continuous values).A few things to consider and remember:TimescaleDB supports the range of PostgreSQL index types (PostgreSQL docs).Creating, altering, or dropping an index on your TimescaleDB hypertable will similarly be propagated to all its constituent chunks (all TimescaleDB hypertables are comprised of many interlinked ""chunk"" tables).Commands made to the hypertable automatically propagate changes down to all of the chunks belonging to that hypertable.To index data, use theCREATE INDEXcommand (e.g.,CREATE INDEX ON conditions (location, time DESC);). You can do this before or after converting the table to a hypertable.By default, TimescaleDB automatically creates a time index on your data when a hypertable is created.And, if thecreate_hypertablecommand specifies an optional ""space partition"" in addition to time (say, thelocationcolumn), TimescaleDB automatically creates the following index:CREATE INDEX ON conditions (location, time DESC);Again, while we encourage new users to use our defaults,the most useful index type varies depending on your data.You can override the defaults when you execute thecreate_hypertablecommand. Prior to doing so, you’ll want to consider things like query patterns, planning, and schemas to understand how different indexes may apply.For indexing columns with discrete values, we suggest using an index like this (using our hypertableconditionsfor the example):CREATE INDEX ON conditions (location, time DESC);For all other types of columns, like columns with continuous values, the index should be in the form:CREATE INDEX ON conditions (time DESC, temperature);Check out our docs fora deeper dive into TimescaleDB best practices for indexing, orreach out on Slackand we’ll happily assist.How should we approach query optimization for a real-time analytical system? We currently have a large Postgres deployment (200G of data), and our system is heavily reliant on Postgres functions. I/O seems to be the one problem that limits the performance of the system.When thinking about this in the context of TimescaleDB, you'll want to consider your chunk size and the amount of memory available to Postgres.Typically, you want to be able to fit a single chunk in memory. If you’re looking for real-time data presentation, make sure you have the right chunk size to ensure that the majority of the data you want to query resides in memory (thus avoiding the I/O bottleneck).For specific guidance, we’d want to understand a bit more about the configuration of your disk subsystem - clearly a topic too deep for this forum, but exactly whatour Slack community is for! Post the details of your scenario, and we’ll work with you to find the right solution.How should I think about partitioning data into multiple dimensions? When is it useful and how do I know which number of partitions to use (e.g., in thenumber_partitionsargument)?Partitioning data into multiple dimensions is most useful when you’re running TimescaleDB in our multi-node setup*, because data is partitioned across nodes.Partitioning data into multiple dimensions may be useful in some cases on a single-node, for instance when you're splitting a hypertable across multiple tablespaces, or when attempting to achieve significantly more data locality (not a common scenario, but can happen).It’s important to note that partitioning data into multiple dimensions is not a replacement for proper indexing. For most developers, indexing on this dimension with a(dimension, time)index is the biggest thing you can do to improve performance.In terms of choosing the number of partitions, it’s usually related to the number of disks/tablespaces or servers that you want the parallelism across.*We’vereleased distributed (multi-node) TimescaleDB as a private betato our community members and customers, and we plan to make an initial version of it more widely available later this year. It will support high write rates, query parallelism, and predicate push down for lower latency, among other capabilities.Contact usto learn how you can join the beta program.View our documentationto learn more about it works.Do you have some info on how to do pattern matching on time-series data?TimescaleDB supports tools and object-relationship mapping that enable you to perform pattern matching.Because Timescale is built on Postgres, you can leverage R, usingRStudioto connect to your Timescale database and operate more advanced “fuzzy matching” functions.Additionally, you can run simpleforloops in Python (or any programming language) to identify pattern matches in your time-series datasets.If you're looking to match strings within the data, Postgres offers an array of matching functions (see docs).We’d also suggestreaching out via Slack, where we and passionate TimescaleDB community members and data experts are active in all channels.Wrapping UpThat concludes this month’s installment of Your Questions Answered, but we’re always here to help - whether you’re just getting started with Timescale or a seasoned community member with a very specific use case.Once again, you can find us at anupcoming Office Hours session,on Slack, or onTwitter.We’re listening!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/your-questions-answered-yqa-april-edition/
2020-05-08T19:36:13.000Z,[New Webinar] Grafana 101 Part II: Getting Started with Alerts,"Jumpstart your monitoring journey with step-by-step demos, queries, and pro tips that’ll get you set up with custom notifications for incidents in your systems, so you can take action immediately.In the first session of our Grafana 101 series, we showed you how to create awesome visualizations to gain insight into real-time performance of your systems, including gauges to track thresholds, single stats to show cache-hit ratios, and more.But, what happens when things go wrong? When something crashes, you’re consuming too much memory, there’s an outage, or users report performance degradation, you need to know about it and take action ASAP.InGuide to Grafana 101 Part II: Getting Started with AlertsI’ll take you from zero to hero in using Grafana to get notified about anomalies, dig into root causes, notify the right teams, and respond to critical issues.RSVPto join me on May 20th at 10am PT / 1pm ET / 4pm GMT.Grafana isn’tonlyfor creating (awesome) visualizationsWhile there are many monitoring systems,Grafanais a great choice. It supports data sources like PostgreSQL, Prometheus, AWS CloudWatch, and many other popular services, and integrates with the communication tools your team already uses (including Slack, OpsGenie, email, and PagerDuty). It’s also open source, making it a popular choice for developers looking for something that’s flexible and cost-effective.Getting setup with alerting can be tricky business. You want to balance alerting on the right metrics and quickly notifying your team, with keeping false-positives down and ensuring you don’t give others notification fatigue.What you’ll learnAs always, I’ll focus on code and step-by-step live demos.We’ll use a scenario where we want to monitor our production database (something we often hear from Timescale customers). I’ll take you through creating and setting alerts based on different rules, like: averages over a period of time, ranges and thresholds – and how to close the loop and send alerts to our team in real-time.More specifically, you’ll:Get an understanding of how alerts work in GrafanaSee how to define key metrics for your scenario and apply them to Grafana’s alerting capabilities*Walkthrough example queries for graphing key metrics and triggering your alertsDefine different alerting rules for specific metrics, including uptime/downtime, average CPU, memory consumption, and total disk usageSetup and receive alerts via various notification channels, like Slack and OpsGenie* (we’ll use metrics critical to monitoring a database, but you may be monitoring a website, Kubernetes cluster, or a larger infrastructure system).A sneak peak at one of the alerts we’ll set up to monitor disk usageWhether you’ve never used Grafana and are looking for a cheaper alternative to proprietary monitoring tools, or are a Grafana pro who’s looking to level up your alerting skills, this session is for you.My goal is that you leave the session with an understanding of when, why, and how to use Grafana alerts and the resources you need to integrate them into your own monitoring setup.I hope to see you there – butsignup even if you’re unable to attend live. I’ll make sure you receive the recording, slides, and resources, as well as answer any questions you have along the way.Myself and other Timescale team experts will be available to answer questions throughout the session (we received 30+ last time!), and share ample resources and technical documentation.See you soon!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grafana-101-alerts-webinar/
2020-04-15T23:23:00.000Z,How to Build More Accurate Grafana Trend Lines: Plot Two Variables with Series-Override,"Plotting multiple variables on one graph and want to compare trends? Learn how to use Grafana’s series-override feature to solve the problem of distorted scale.Problem: Skewed Trends Due to Differences in Data ScaleMany times, we want to plot two variables on the same graph (a useful feature of viz tools likeGrafana), but, run into one big problem: the scale of one of the variables distorts the trend line of the other variable.Case in point, this is a graph I put together to track COVID-19 cases and deaths in the USA:The scale of total_cases makes the total_deaths trend line look flat, when it's actually growingAs you can see, the scale of the total cases makes the trend line for deaths look flat, even though it’s actually growing rapidly, as we can see from the graph which plots only COVID-19-related deaths:USA COVID-19 reported deaths over time (with a more obvious upward trend line)Viewing two related data points in one graph is extremely useful to create informationally dense dashboards and compare related variables, butdistorted trends can have large consequences- whether we view the COVID fatality situation more optimistically than we should, or if we inaccurately compare our eCommerce site’s unique visitors relative to session crashes.We need a way to more accurately represent the trends of both variables, while still plotting them on the same axis.Solution: Two Y Axes!The solution is to use a different Y axis for each variable on our graph. Continuing with my COVID-19 example, this means one for the total cases variable and one for the total deaths variable, as shown in the graph below:Here, we use two Y axes: one for COVID-19 total cases, on the left, and one for total deaths, on the rightEach axis has their own scale, allowing us to more accurately see the growth of each trend line without the scale of one variable (e.g., total volume of reported cases) impacting how another variable (e.g., growing number deaths) appears.Try it yourself: Implementation in Grafana with Series OverrideIn this post, I'll show you how to use Grafana’s series override feature to implement two Y axes (and, thus, solve our two-trend line problem).We’ll use the example of charting the spread of COVID-19 cases and deaths in the USA, but the concepts apply to any dataset you’d like to visualize in Grafana. We’ll get our COVID-19 data fromthe New York Times’  public dataset.PrerequisitesTo replicate the graph I’ll create in the following steps, you’ll need a:Grafana instanceTimescaleDB database, loaded with the NYT COVID-19 data.PostgreSQL datasource, with TimescaleDB enabled, connected to your Grafana instance. Seehereto get this setup.Grafana panel with Graph visualization using the PostgreSQL database with the COVID data as the data source.Step 1: Create two seriesPlotting multiple series in one panel is a handy Grafana feature. Let’s create two series, one for COVID-19 cases and the other for COVID-19 deaths:SELECT date as ""time"", sum (cases) as total_cases, sum (deaths) as total_deaths
FROM states
GROUP BY time
ORDER BY time;Notice: we alias ""total cases"" and ""deaths"" as total_cases and total_deaths, respectively.Step 2: Modify our visualization to add a second Y axisGrafana Visualization settings for adding a second Y-Axis to our graphFirst, navigate to the visualization panel (pictured above) and select theAdd series overridebutton.Then, we select the name of the series we'd like to override, “total_deaths” from the drop down menu. Then, to associate the series with the second Y axis, we select the ‘plus’ button and then select Y-Axis 2, as shown below:How to find the Y-Axis 2 optionWhen we navigate down to the Axes section, we seeLeft YandRight Y, where we customize the units and scale for each axis.In our case, we’ll leave the units asshortand the scale aslinear, since those defaults work for the scalar quantities in our COVID dataset.Finally, we save the graph and refresh. We should now see both variables, total cases and deaths, plotted on the same graph, but with differently scaled axes.Trend line before (singular axis) and after (two axes).Notice: we more clearly see how quickly COVID-19 deaths in the USA are growing, which was difficult to discern in the original graph (where deaths were plotted with total COVID-19 cases on the same Y axis).That’s it! We’ve successfully created a graph with two Y axes, using series-override 🎉.Learn MoreFound this tutorial useful? Here are two more resources to help you build Grafana dashboards like a pro:Grafana WebinarJoin me on April 22nd at 10am PT/1pm ET/4pm GMTwhere I’ll demo how to:Create 5+ different visualizations for data from IoT, infrastructure monitoring, and public datasetsCombine various data types, including geo-spatial and time-series data, in our dashboardsTake my demo and customize it for your project, team, or organizationI’ll focus on code and step-by-step live demos – and I and my dashboarding expert colleagues will be available to answer questions throughout the session, plus share ample resources and technical documentation.Signup even if you’re unable to attend live, and I’ll make sure you receive the recording, slides, and resources - and answer any questions you may have along the way.All-in-One Grafana TutorialWe’ve compiled all our tutorials, tips, and tricks for visualizing PostgreSQL data in Grafana into this one doc.You’ll find everything from how to create visuals for Prometheus metrics to how to visualize geo-spatial data using a World Map.Check it outhere.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grafana-series-override/
2020-03-23T16:50:26.000Z,"Homegrown Monitoring for Personal Projects: Prometheus, TimescaleDB, and Grafana FTW","Learn how Tyler - one of our newest additions to the team - setup a ""roll-your-own monitoring"" solution with 99% storage savings – and how you can do the same, be it for your personal projects or a critical piece ofbusinessinfrastructure.Like many of you, I love to tinker. I overcomplicate the things in and around my home because, while I don’tNEEDan enterprise-ready network, it’s fun to play with anyway. I’ve got anOwncloudinstance I run for a friend, api-hole, a lab Postgres instance, a CIFS server for backing up thedashcam and sentry footage from my Tesla, and a Docker host for running various miscellany (including all the bits for this project!), plus other assorted things.As a new addition to the Timescale team, I decided I needed a good dogfood project, and monitoring all of these things felt like a good way to get started. So, I did some reading and decided to set upPrometheus,TimescaleDB(+ theTimescale Prometheus Adapter), andGrafanato get acquainted with the various pieces.I hoped to get a better sense of how our customers might be using our software, as well as just get more familiar with TimescaleDB, Postgres, Grafana – and how they all fit together. Plus, it’d be good to know if my Owncloud server was running out of storage, or if that pi-hole was running out of RAM.My SetupPrometheus was straight-forward for me to get up and going; their focus on discrete parts doing one thing and doing them well really shines.The Prometheus server and the exporters are both hassle-free: run the exporter on the host you want to monitor, point the prometheus server at the right host and port, and you're done!The node-exporter is lightweight, and the settings for Prometheus itself are likewise minimal, making for a quick and easy solution.If you’d like to tinker along with me, I’ve got an example of what I’m using for the monitoring bitson GitHub(PRs always welcome!). Did I mention I like to overcomplicate things? :)In any event, I’m gathering Prometheus node-exporter data from 4 hosts, and Prometheus pi-hole-exporter data from the pi-hole in addition.Here’s a look at what this looks like, with different parts at two locations(some at my office, and some at home).Quick sketch of my architecture setupThis monitoring setup generates about 3.5GB of raw data every day - not a lot in terms of monitoring for a business, but quite a lot for a side project! Since this is just for some messing around, I don’t want to have to put a bunch of disk storage behind it.Here's the monitoring view, usinga great dashboardby starsL.Node-Exporter Monitoring DashboardWhat to do with my data?I considered dropping data older than a few days, but before I did that, I thought I should check outTimescaleDB’s native compression.My reasoning: the more data you can collect from your systems, the better you can see trends, plan for growth, and understand ideal service windows for maintenance. Not super important for home stuff, but in the interest of making this as practical as possible, I wanted the full picture of the insights I could glean. To do that, I needed to grab and retain as much data as I reasonably could.Enabling compression? Just as straightforward as getting Prometheus up and running 🎉:ALTER TABLE metrics_values 
SET (timescaledb.compress, timescaledb.compress_orderby = 'labels_id, time DESC');Then, I set my policy to compress data older than 3 days:SELECT add_compress_chunks_policy
('metrics_values', INTERVAL '3 days');I’ve let this run for a couple weeks now, and the results have been astounding. I figured that I’d be able to keep roughly two weeks worth of data before rolling it off – but the compression on my Prometheus data is so effective,I’ll be able to keep months worth of data without using any significant storage(1.5 GB/month rather than about 105GB/month). This lets me see much longer usage patterns.See for yourself:SELECT uncompressed_total_bytes, compressed_total_bytes 
FROM timescaledb_information.compressed_hypertable_stats;uncompressed_total_bytescompressed_total_bytes58 GB637 MBAdmittedly, this is just a single sample with a relatively small amount of data, but I’m seeing compression coming in at roughly 99% space savings (approximately 58000MB →  637MB) for various kinds of Prometheus data (no small feat!).If you want more information about how compression works and the benchmarks we saw during internal testing, check out thisblog post.If you want to try it out yourself and see how much your data compresses,our Quick Start documentationis a great place to start.If you have questions, need help troubleshooting, or want to get some recommendations,reach out to us on Slack at any time.There’s always at least one Timescale Engineer - including myself - “on duty” and community members are keen to jump in and help too.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/homegrown-monitoring-for-personal-projects-prometheus-timescaledb-and-grafana-ftw/
2020-03-05T19:11:10.000Z,Charting the Spread of COVID-19 Using Data,"Data gives us insight into the world around us – and allows us to visualize and prepare for the spread of the COVID-19 pandemic.In recent months, we’ve seen thespread of the COVID-19 virus(commonly referred to as ""coronavirus""). Not only are we a globally distributed team, some Timescale team family members see fears and concerns first-hand while working in our emergency rooms.First and foremost, our thoughts are with those who've been affected in any way. We join the global community in hoping for the best in the coming weeks and months, especially in containing transmission and reducing the virus' spread.We were recently made aware of atime-series datasetfrom Johns Hopkins University, containing daily case reports from around the world. Timescale community memberJoel Natividadcreated thisrepo of time-series utilitiesthat enables people to use the Johns Hopkins University dataset with TimescaleDB to better monitor the COVID-19 situation.In this post, we'll walkthrough how to load the dataset into TimescaleDB and use Grafana to visualize queries. Our hope is that you’ll be able to use this dataset and tutorial as a starting point for your own analysis (extending upon the queries we include), or to share information with colleagues, family, and friends.You can getTimescaleDBin two forms: Managed Service for TimescaleDB (with free credits) or TimescaleDB for download. You canfollow these instructionsto choose your installation method. If you haven’t done so, you’ll also want toinstall psqlto access the database from your command line.Ingesting the datasetOnce you’ve installed TimescaleDB, you will need to load the dataset. First, you will want to clone the GitHub repository Joel created:git clone https://github.com/dathere/covid19-time-series-utilities.gitThis repository contains several files, which are explained in theREADME. Setup your database and ingest the dataaccording to the instructionsin the GitHub repository.Once the data is fully ingested, you should be able to login to your database frompsql:psql -x ""postgres://[YOUR USERNAME]:[YOUR PASSWORD]@[YOUR HOST]:[YOUR PORT]/[DB_NAME]?sslmode=require""Once you’re logged in, you will be able to run the following query:SELECT * FROM covid19_ts LIMIT 5;Your output should look something like this:province_state | country_region |    observation_date    | confirmed | deaths | recovered
---------------+----------------+------------------------+-----------+--------+-----------
Hubei          | Mainland China | 2020-01-31 23:59:00+00 |      5806 |    204 |       141
Zhejiang       | Mainland China | 2020-01-31 23:59:00+00 |       538 | [null] |        14
Guangdong      | Mainland China | 2020-01-31 23:59:00+00 |       436 | [null] |        11
Henan          | Mainland China | 2020-01-31 23:59:00+00 |       352 |      2 |         3
Hunan          | Mainland China | 2020-01-31 23:59:00+00 |       332 | [null] |         2
(5 rows)At this point, your time-series database is properly configured and you've loaded the COVID-19 dataset.Note: the Johns Hopkins University data is a running total, not a per-day total. So, the data for February 23rd for any given location represents a cumulative tally of all cases in that location as of February 23rd.Before we continue, we need to clean up our dataset.In some cases, a given row in thecovid19_tsorcovid19_locationstable may not have aprovince_stateentered properly. Cleaning public datasets is fairly common, and fortunately, this dataset is easy to prepare for the rest of our tutorial. Inpsqlenter the following commands:UPDATE covid19_locations 
SET province_state = 
    CASE WHEN province_state = '' 
    THEN country_region 
    ELSE province_state 
    END;At this point, all our rows have either a specific Province or State in theprovince_statecolumn, or, failing that, we default to thecountry_region.Now, let’s start running queries that will give us some insight into the pandemic.How many confirmed cases were reported worldwide each day?Let’s start by identifying how many cumulative cases have been identified each day; the dataset starts on January 22nd, so that’s where we'll start for our query.SELECT time_bucket('1 day', observation_date) AS day,
       SUM(confirmed)
FROM covid19_ts
GROUP BY day
ORDER BY day;Your output should look something like this:day           |  sum
-----------------------+-------
2020-01-22 00:00:00+00 |   555
2020-01-23 00:00:00+00 |   653
2020-01-24 00:00:00+00 |   941
2020-01-25 00:00:00+00 |  1438
2020-01-26 00:00:00+00 |  2118
2020-01-27 00:00:00+00 |  2927
2020-01-28 00:00:00+00 |  5578
2020-01-29 00:00:00+00 |  6165
2020-01-30 00:00:00+00 |  8235
2020-01-31 00:00:00+00 | 10037
2020-02-01 00:00:00+00 | 12009
2020-02-02 00:00:00+00 | 16685
2020-02-03 00:00:00+00 | 19719
2020-02-04 00:00:00+00 | 23821
2020-02-05 00:00:00+00 | 27457
(edited for length)How many confirmed cases have been reported each day near Milan, Italy?As of the publication date of this post, northern Italy has the highest concentration of COVID-19 cases, outside of China, and it continues to intensify.What if we wanted to see the growth rate of COVID-19 in Italy?To do this, we query on thecountry_regionfield in our database:SELECT * FROM covid19_ts WHERE country_region = 'Italy';Our output will look something like this:province_state | country_region | observation_date       | confirmed | deaths | recovered
---------------+----------------+------------------------+-----------+--------+-----------
               | Italy          | 2020-01-31 23:59:00+00 |         2 | [null] |    [null]
               | Italy          | 2020-01-31 08:15:00+00 |         2 |      0 |         0
               | Italy          | 2020-01-31 08:15:53+00 |         2 |      0 |         0
               | Italy          | 2020-02-07 17:53:02+00 |         3 |      0 |         0
               | Italy          | 2020-02-21 23:33:06+00 |        20 |      1 |         0
               | Italy          | 2020-02-22 23:43:02+00 |        62 |      2 |         1
               | Italy          | 2020-02-23 23:43:02+00 |       155 |      3 |         2
               | Italy          | 2020-02-24 23:43:01+00 |       229 |      7 |         1
               | Italy          | 2020-02-25 18:55:32+00 |       322 |     10 |         1
               | Italy          | 2020-02-26 23:43:03+00 |       453 |     12 |         3
               | Italy          | 2020-02-27 23:23:02+00 |       655 |     17 |        45
               | Italy          | 2020-02-28 20:13:09+00 |       888 |     21 |        46
               | Italy          | 2020-02-29 18:03:05+00 |      1128 |     29 |        46
               | Italy          | 2020-03-01 23:23:02+00 |      1694 |     34 |        83
               | Italy          | 2020-03-02 20:23:16+00 |      2036 |     52 |       149
(15 rows)How many confirmed cases have been reported each week near me?What if we wanted to look at cases near a specific location?This requires that we make use of thelatitudeandlongitudeinformation stored in our database. To use the location, we’ll need to get our tables ready for geospatial queries.One caveat: The Johns Hopkins University dataset has limited information about specific locations. All cases in Italy are marked as “Italy,” for example, while cases in China and the United States may be more specific with regional or city locations.We can use the PostGIS PostgreSQL extension to help analyze geospatial data. (This is available out-of-the-box onManaged Service forTimescaleDB, or you canmanually installfor self-hosted versions.) PostGIS allows us to slice data by time and location within TimescaleDB.CREATE EXTENSION postgis;Then, run the\dxcommand inpsqlto verify that PostGIS was installed properly. You should see the PostGIS extension in your extension list, as noted below:List of installed extensions
Name        | Version |   Schema   |                             Description
------------+---------+------------+---------------------------------------------------------------------
plpgsql     | 1.0     | pg_catalog | PL/pgSQL procedural language
postgis     | 2.5.3   | public     | PostGIS geometry, geography, and raster spatial types and functions
timescaledb | 1.6.0   | public     | Enables scalable inserts and complex queries for time-series data
(3 rows)Thecovid19_locationstable in our database includes the locations in the Johns Hopkins University dataset, as well as their latitude and longitude coordinates.For example, we can see all the locations in the United States like so:SELECT * FROM covid19_locations WHERE country_region = 'US' LIMIT 10;The output for this query looks something like this:province_state                | country_region | latitude | longitude 
---------------------------------------------+----------------+----------+-----------
 Unassigned Location (From Diamond Princess) | US             |  35.4437 |  139.6380
 King County, WA                             | US             |  47.5480 | -121.9836
 Santa Clara, CA                             | US             |  37.3541 | -121.9552
 Snohomish County, WA                        | US             |  48.0330 | -121.8339
 Cook County, IL                             | US             |  41.7377 |  -87.6976
 Fulton County, GA                           | US             |  33.8034 |  -84.3963
 Grafton County, NH                          | US             |  43.9088 |  -71.8260
 Hillsborough, FL                            | US             |  27.9904 |  -82.3018
 Providence, RI                              | US             |  41.8240 |  -71.4128
 Sacramento County, CA                       | US             |  38.4747 | -121.3542
(10 rows)We need to alter thecovid19_locationstable to work with PostGIS.To start, we’ll add geometry columns for case locations:ALTER TABLE covid19_locations
    ADD COLUMN location_geom geometry(POINT, 2163);Next we’ll need to convert the latitude and longitude points into geometry coordinates (so they play well with PostGIS).UPDATE covid19_locations
    SET location_geom = ST_Transform(ST_SetSRID(ST_MakePoint(longitude, latitude), 4326), 2163);Lastly, let’s say we want to examine cases near Seattle, in the United States. Seattle is located at (lat, long) (47.6062,-122.3321).Our database is setup, so we're ready to run a geospatial query that returns the number of confirmed COVID-19 cases within 75km of Seattle, each day, since the start of our dataset. (TimescaleDB is based on PostgreSQL, so we can use all our favorite SQL functions.)In this case, we need to use a subquery for the information in ourcovid19_locationstable and a broader query for all confirmed cases that match the locations in our subquery.First, we write the subquery by querying thecovid19_locationstable for all locations within 75km of Seattle:SELECT *
FROM covid19_locations
WHERE ST_Distance(location_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-122.3321, 47.6062), 4326), 2163)) < 75000;We should get output that looks like this:province_state       | country_region | latitude | longitude |                   location_geom
---------------------+----------------+----------+-----------+----------------------------------------------------
King County, WA      | US             |  47.5480 | -121.9836 | 010100002073080000A2B8C609FEC838C1004A6D25430F1F41
Snohomish County, WA | US             |  48.0330 | -121.8339 | 010100002073080000344A7104C26438C1768752D481082141
(2 rows)Now, we need to find all confirmed cases in ourcovid19_tsdataset whoseprovince_statematches the result of our first query.SELECT time_bucket('1 day', observation_date) as day,
       SUM(confirmed) AS near_sea
FROM covid19_ts
WHERE province_state IN (
    SELECT province_state
    FROM covid19_locations
    WHERE ST_Distance(location_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-122.3321, 47.6062), 4326), 2163)) < 75000
)
GROUP BY day
ORDER BY day;Your output should look something like this:day           | near_sea 
------------------------+----------
 2020-02-29 00:00:00+00 |        1
 2020-03-01 00:00:00+00 |        2
 2020-03-02 00:00:00+00 |       18
 2020-03-03 00:00:00+00 |       27
 2020-03-04 00:00:00+00 |       39
(5 rows)Use Grafana to visualize confirmed cases by geographic locationWe have the dataset, we have our queries, and, now, let's take the COVID-19 dataset and visualize the total number of confirmed cases by geographic location.To create our visualization, we’ll use Grafana. Follow ourshort tutorial on getting started with Grafanato setup Grafana in Managed Service for TimescaleDB, your desktop, or using Grafana Cloud, using the dataset you’ve been using in this post.In Grafana, create a new dashboard and add a new visualization. In the visualization menu, search for and select the WorldMap panel type.To build our visualization, we will need our information organized properly in our query:For each dayA sum of all confirmed casesGrouped by theprovince_statewhere they happenedPlotted on the map using the latitude and longitude for theprovince_stateOrdered by timeTo build this query, we use a SQLINNER JOINstatement to combine information from thecovid19_tsdataset and thecovid19_locationstables:SELECT time_bucket('1 day', covid19_ts.observation_date) AS day,
       SUM(covid19_ts.confirmed) AS confirmed, 
       covid19_ts.province_state as location, 
       covid19_locations.latitude AS latitude, 
       covid19_locations.longitude AS longitude
FROM covid19_ts
INNER JOIN covid19_locations ON covid19_locations.province_state = covid19_ts.province_state 
WHERE $__timeFilter(covid19_ts.observation_date)
GROUP BY day,
         confirmed,
         location,
         latitude,
         longitude 
ORDER BY day;To enter this query in Grafana, click “Edit SQL” and type it in manually (or copy-paste from the above):And then in the WorldMap configuration, we set the field mapping to match our query results:And, once completed, we see the impact of the COVID-19 pandemic on the Grafana WorldMap panel:Spread of COVID-19 globally, visualized in GrafanaHow rapidly is COVID-19 spreading near me?As mentioned earlier, the Johns Hopkins dataset includes cumulative data. For example, an entry for March 2, 2020 contains all cases that have beenconfirmedat that point in time. Suppose we wanted to learn the rate at which cases have been identified near a specific location.TimescaleDB includes a feature calledcontinuous aggregates. A continuous aggregate recomputes a query automatically at user-specified time intervals and maintains the results into a table. Thus, instead of everyone running an aggregation query each time, the database can run a common aggregation periodically in the background, and users can query the results of the aggregation. Continuous aggregates should improve database performance and query speed for common calculations.In our case, we want to maintain a continuous aggregation for the daily change in confirmed cases. Let's look at this continuous aggregation query:CREATE VIEW daily_change
WITH (timescaledb.continuous, timescaledb.refresh_lag = '-2 days')
AS
SELECT
  country_region,
  province_state,
  time_bucket('2 days', observation_date) as bucket,
  first(confirmed, observation_date) as yesterday,
  last(confirmed, observation_date) as today,
  last(confirmed, observation_date) - first(confirmed, observation_date) as change
FROM
  covid19_ts
GROUP BY country_region, province_state, bucket;The first line of this query creates a continuous aggregation nameddaily_change. We then select atime_bucketwith an interval of two days calculated using thecovid19_ts.observation_datefield.In our table, we'll create ayesterdayandtodaycolumn, as well as achangecolumn that represents the delta between the two.Under normal circumstances with large amounts of data, TimescaleDB calculates continuous aggregates in the background, but if you want to see the result immediately, you can force it:ALTER VIEW daily_change SET (timescaledb.refresh_interval = '1 hour');
ALTER VIEW daily_change SET (timescaledb.refresh_lag = '-2 days');
REFRESH MATERIALIZED VIEW daily_change;We can run this continuous aggregation with a simple SQL query:SELECT * FROM daily_change;The result of that query should look something like this:day           | yesterday | today  | change |                  location                   
------------------------+-----------+--------+--------+---------------------------------------------
 2020-01-22 00:00:00+00 |         2 | [null] | [null] | 
 2020-01-22 00:00:00+00 |         1 |      9 |      8 | Anhui
 2020-01-22 00:00:00+00 |    [null] | [null] | [null] | Australia
 2020-01-22 00:00:00+00 |        14 |     22 |      8 | Beijing
 2020-01-22 00:00:00+00 |    [null] | [null] | [null] | Brazil
 2020-01-22 00:00:00+00 |         6 |      9 |      3 | Chongqing
 2020-01-22 00:00:00+00 |    [null] | [null] | [null] | Colombia
 2020-01-22 00:00:00+00 |         1 |      5 |      4 | Fujian
 2020-01-22 00:00:00+00 |    [null] |      2 | [null] | Gansu
 2020-01-22 00:00:00+00 |        26 |     32 |      6 | GuangdongWe can also incorporate the continuous aggregation in other queries, such as the one we used earlier to see the rate of change in confirmed cases near Seattle, Washington.SELECT bucket, 
       yesterday, 
       today, 
       change, 
       province_state, 
       country_region
FROM daily_change
WHERE province_state IN (
    SELECT province_state
    FROM covid19_locations
    WHERE ST_Distance(location_geom, ST_Transform(ST_SetSRID(ST_MakePoint(-122.3321, 47.6062), 4326), 2163)) < 75000
)
GROUP BY bucket, 
         yesterday, 
         today,  
         change, 
         province_state, 
         country_region
ORDER BY bucket;Our output should look like this:bucket         | yesterday | today | change |    province_state    | country_region 
------------------------+-----------+-------+--------+----------------------+----------------
 2020-02-29 00:00:00+00 |         1 |     2 |      1 | Snohomish County, WA | US
 2020-03-02 00:00:00+00 |         4 |     6 |      2 | Snohomish County, WA | US
 2020-03-02 00:00:00+00 |        14 |    21 |      7 | King County, WA      | US
 2020-03-04 00:00:00+00 |         8 |     8 |      0 | Snohomish County, WA | US
 2020-03-04 00:00:00+00 |        31 |    31 |      0 | King County, WA      | US
(5 rows)ConclusionData gives us insight into the world around us and understanding data enables us to communicate these insights to many people. We encourage you to explore the dataset and share your dashboards and insights on Twitter using#Covid19Data(@-mention@TimescaleDBif you have questions or comments).If you need help using TimescaleDB or writing visualizations of your own, follow theHello, Timescale! tutorial. And,join our Slackto ask questions and get help from the global community of Timescale users (our engineering team is also active on all channels).Finally, no matter where you are, take prudent actions to prepare yourself and your family. Please follow the guidelines and instructionsfrom the World Health Organization.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/charting-the-spread-of-covid-19-using-timescale/
2020-06-03T22:46:30.000Z,“Grafana 101: Getting Started with Alerting” Recap and Resources,"Get step-by-step demos and learn how to define 3 different alert rules and send notifications through channels like Slack, PagerDuty, and OpsGenieWe recently finished part two of our “Grafana 101” webinar series, a set of technical sessions designed to get you up and running withGrafana’svarious capabilities - complete with step-by-step demos, tips, and resources to recreate what we cover.In this one, we focus on “Getting Started with Alerts,"" where I go through what alerting in Grafana entails, show you how to select and set up 3 common alerts for key metrics, and create triggers to send notifications through popular channels. And, to make sure you leave ready to set up your own alerting and monitoring systems, I share various best practices and things I’ve learned along the way.We know not everyone could make it live, so we’ve published therecordingandslidesfor anyone and everyone to access at any time.If you missed this one, or are keen to level up your Grafana skills, we’re hosting our 3rd session, “Guide to Grafana 101: Getting Started with Interactivity, Templating and Sharing” on June 17th, 2020 (RSVP here)Missed the session? Don't worry, here's the recording!What you’ll learnAlerting is a crucial part of any monitoring setup. But, getting them set up is often tricky and time consuming, especially if you’re dealing with multiple data sources.Thankfully, you can configure your visualizations and alerts for the metrics you care about in the same place, thanks to Grafana’salerting functionality!While Grafana may be best known for its visualization capabilities, it’s also a powerful alerting tool. Personally, I like using it to notify me about anomalies, because it saves me the overhead of adding another piece of software to my stack – and I know many community members feel the same.I break the session into four parts:Alerting PrinciplesAlerts tell us when things go wrong and get humans to take action.When you implement alerts in any scenario, there are two important universal best practices:Avoid over-alerting: If an engineer gets an alert too frequently, it ceases to be useful or serve its purpose (i.e., instead of responding quickly, people will quickly tune them out as noise).Select use-case specific alerts: Different scenarios require monitoring different metrics, so alerts for monitoring a SaaS platform (site uptime, latency, etc.) are different than an alerts setup for monitoring infrastructure (cluster health, disk usage, CPU/memory use).Alerts in GrafanaIn this section, I cover how alerts work in Grafana and their two constituent parts: alert rules and notification channels. Note: Grafana’s alerting functionality only works for graph panels with time-series output. At first this may seem limiting, but, thankfully, it isn’t for two reasons:You can convert any panel type (e.g., gauge, single stat, etc.) into a graph panel and then set up your alerts accordingly.Your graphs must have the output format of ‘time-series’, which is a reasonable constraint: you want to monitor how a certain metric changes over time, so your data is inherently time-series data.You’ll also learn about the anatomy of alert rules and conditions, see how theFORparameter works, and understand the various states alerts can take, depending on whether their associated alert rules areTRUEorFALSE.Alerts only work on graph panels with time-series output in GrafanaLet’s Code: 3 Alerts and 3 Notification channelsAfter seeing the basics, we jump into the fun part: creating and testing alerts!Using the scenario of monitoring a productionTimescaleDBdatabase, we set up different types of alerts for common monitoring metrics and connect our alert monitoring to popular notification channels:Alert Type:Alerts usingFORMetric: Sustained high memory usageNotification channel: sent via Slack, where we have a channel to notify our DevOps team about new alerts from our Grafana setup.Alert Type:Alerts without FORMetric: Disk usageNotification channel: sent via PagerDuty, where an incident is automatically created and relevant DevOps teams and support personnel are notified (according to our pre-configured PagerDuty escalation policies).Alert Type: Alerts withNO DATAMetric: Database alivenessNotification channel: sent via OpsGenie, where an alert is created and sent to the DevOps team and other support personnel (according to the notification policies we’ve configured for our team in OpsGenie). We'll use Slack as an additional notification channel as well.Alerts using FORThis part of the demo shows how to define an alert for sustained high memory usage on the database, using the Grafana alerting parameterFOR. The parameterFORspecifies the amount of timeforwhich an alert rule must be true before the ALERTING state is triggered and an alert is sent via a notification channel.UsingFORis common for many alerting scenarios, as you often want to wait for your alert rule to be true for a period of time in order to avoid false positives (and waking people up in the middle of the night without cause).Once we’ve defined our alert rule in Grafana, I show you how to set up Slack as a notification channel, so alert messages reach you (or the right team members) in a timely manner. You’ll see how to customize the message body with pertinent info and send notifications to specific channels and mention team members.We set up Slack as a notification channel to show alerts from our Grafana dashboardI also take you through howFORalerts work in Grafana, using a state transition diagram to give you a mental model for when, how, and why to use them.See step-by-step lifecycles of alert states that an alert usingFORcould takeAlerts without FORThis second part of the demo shows how to use a simple threshold condition to define an alert for high disk volume; this alert rule doesn’t use theFORparameter, so alerts are sent as soon as the rule is triggered. In this example, there’s no need to to use theFORparameter, since disk usage (the metric we’re alerting on) doesn’t fluctuate up and down with time and usually only increases. Therefore, we can send out an alert as soon as our alert condition isTRUE.Once we’ve defined our alert rule, I show you how to connect Grafana toPagerDuty, so that alerts in Grafana create cases in PagerDuty and notify teams via phone, email, or text, based on your PagerDuty configurations (e.g., whatever rules and notification methods you’ve set up in PagerDuty).We setup PagerDuty as a notification channel to show alerts from our Grafana dashboardAs in the first example, I use a state transition diagram to help you visualize how this works and when you’d use it versus other types.See step-by-step lifecycles of alert states that an alertwithoutFORcould takeAlerts with NO DATAThe final part of the demo shows how to define an alert for “aliveness” on our database, so we know if our database is up or down. Like our high disk volume alert, we use a threshold condition, and to show you to triggerNO DATAalerts, I turn my demo database off to simulate an outage/downtime.This immediately triggersNO DATAalerts for other alert rules on metrics from the database, namely sustained high memory and disk usage, the two alert rules we set up earlier in the demo.NO DATAalerts rules are useful to distinguish between an alert rule being true and there being no data with which to evaluate the alert rule. You would useALERTINGas the state for the former condition andNO DATAas the state for the latter. UsuallyNO DATAalert rules indicate that there is a problem with the data source (e.g., the data source is down or has lost connection to Grafana).We setup OpsGenie and Slack as a notification channels to show alerts from our Grafana dashboardFrom there, I show you how to connect Grafana toOpsGenie, so that alerts in Grafana create cases and alerts in OpsGenie, which can then notify various teams, using methods like email, text, and phone call. I also show how to send notifications via our existing Slack notification channel.And, of course, I share a diagram for this one too :).See step-by-step lifecycles of alert states that an alert withNODATAcould takeResources and Q+AWant to recreate the sample monitoring setup shown in the demo? Or perhaps you want to modify it to use your data sources and notification channels to set up your own monitoring system? No worries, we have you covered!We link to several resources and tutorials to get you on your way to monitoring mastery:Reason about Grafana Alert States with this handy reference chartSummary of all Grafana alert state transitionsReplicate our demo by following ourGrafana alerting tutorialFollow along with thesession recordingUse ourPrometheus Adapter and Helm Chartsto install Grafana, TimescaleDB and Prometheus in your Kubernetes cluster, with one line of code.Get started with Timescale Cloud(our hosted time-series database, which comes with a 30-day free trial)Join our Slackto ask questions and get Grafana help from Timescale engineers - including Grafana contributors - and our helpful developer community.Community QuestionsHere’s a selection of questions we received (and answered) during the session:Q: Does alerting work with templating or parameterized queries?A: Template variables in Grafana do not work when used in alerts.  As a workaround, you could set up a custom graph that only displays the output of a specific value that you’re templating against (e.g., a specific server name) and then define alerts for that graph.If you use Prometheus, PrometheusAlertmanagermight be worth exploring as well.Alternatively you can explore using wildcards in your queries to mimic the effect of templating. See the following Grafana issues for more information:Issue 1,Issue 2Q: If we setFORto 5min, we wait for 5mins to go fromPENDINGtoALERTING. Do we wait for the same period of time to transition back fromALERTINGtoOK?A: No, theFORparameter defines the amount of time the alert rule must be in theTRUEstate before an alert is sent.TheFORparameter applies to transitions fromPENDINGtoALERTINGonly. You don’t need to wait for theFORperiod of time to go fromALERTINGtoOK.As soon as the alert rule evaluatesFALSE(i.e., the issue is resolved), the alert state will change fromALERTINGtoOKand be re-evaluated the next time your alert job runs.Q: When specifying the condition to use in your alert rule, what does “now” mean? Why do you select to alert on Query A from 5 minutes ago till now? Isn't italwaysnow?A:  Grafana requires you to have a period of time to evaluate if alerts need to be sent. This time period is the period over which you want to do calculations on, which is why many conditions have aggregate functions likeAVG,MAX,DIFF,LAST, etc.So, when we sayQuery(A, 5m, now), it means that we evaluate the alert rule on Query A from 5m ago to now.  You can also set alert rules that sayQuery(A, 6m, now-1min)to account for a lag in data being inserted in your data source or network latency.This is important, because if an alert condition usesQuery(A, 5m, now)and there is no data available fornow,NO DATAalerts will fire (i.e., there isn’t any data with which to evaluate the rule).You’ll want to select a time interval to evaluate your aggregate function that’s greater (longer) than the time period in which you add new data to the graph that’s triggering your alerts.For example, in my demo, we scrape our database every 10 seconds. If we set an alert rule likeQuery(A, 5s, now), we’d end up gettingNO DATAalert states constantly, because our data is not that fine grained.Ready for more Grafana goodness?Thanks to those who joined me live! To those who couldn’t make it, myself and the rest of Team Timescale are here to help at any time.Reach out via our Slackand we’ll happily assist!For easy reference, here are therecordingandslidesfor you to check out, re-watch and share with teammates.Want to level up your Grafana skills even more? Sign up for session 3 in our Grafana 101 technical series on Wednesday, June 17: “Guide to Grafana 101: Getting Started with Interactivity, Templating and Sharing”🔍RSVPto learn about making interactive, re-usable dashboards that you can version control and collaborate on with teammates!To learn about future sessions and get updates about new content, releases, and other technical content, subscribe to ourBiweekly Newsletter.Excited to see you at the next session!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grafana-101-getting-started-with-alerting-recap-and-resources/
2020-04-20T14:00:00.000Z,"How to Use Grafana Variables to Make More Interactive Dashboard Visualizations (Aka a Lot Less Boring, a Lot More Useful)","Learn how to add features that allow you, your teammates, and your stakeholders to drill into specific details, see all results, and quickly get the info you need.The (all too common) Problem: Boring, kind of useful, static dashboardsThose of us that work with data often want to make useful dashboards that make it easier for ourselves and other people within our team and organization to make sense and get insight about the data we collect.A common problem I’ve run into (both when creating dashboards and using them as a stakeholder) is that many dashboards aren’t interactive enough for non-technical stakeholders to answer their questions without asking engineers to write new code or change the underlying queries powering the dashboard.Or worse, stakeholders try to dig into the code and accidentally break things...Solution: Make your dashboards interactive (and user-friendly!)Fortunately, many visualization tools have features to make your graphs, maps, tables, and other visualizations interactive via the tool's native UI. It’s a win-win situation: more usability for the stakeholders who rely on your dashboards, and less of your time spent attending to minor customization changes.Here’s an example inGrafana, an open-source visualization tool, where I’ve created “pickers” (filters) that allow me and others to choose values from a drop down menu and immediately see our selections appear in the visual -without having to change any of the underlying SQL queries powering the dashboard.Grafana variables allow you to use a drop down menu to select various options, no code modifications required.However, enabling features like pickers in the Grafana UI can be tricky.In the rest of this post, I'll show you how to use Grafana’svariablesfeature to build your own interactive dashboards. I’ll use the example of monitoring the live locations of buses going on different routes in New York City to illustrate, but the steps I follow will work for any scenario.Try it yourself: Implementation in GrafanaReady to learn how to use variables in Grafana dashboards, powered by PostgreSQL queries?Pre-requisites:Grafana instancePostgreSQL datasource with TimescaleDB enabled, connected to your Grafana instance. Seeherefor how to connect one.If you don't knowTimescaleDB, it's an extension that will make your PostgreSQL queries (and visualizations) much faster.Click here to learn more.I’ll use the example of visualizing the real-time location in New York City, using data from the Metropolitan Transportation Authority.I have an existing Grafana World Map panel setup, pictured below. To replicate my initial setup (I use PostgreSQL with TimescaleDB  enabled as my datasource), you can clone and follow the steps in thisGitHub repo.Panel with visualization, using PostgreSQL as the data source.Here’s what my panel looks like before we make it interactive.You can download the JSON to replicate the dashboard in thisGitHub repo.Initial World Map panel, showing live locations of buses in New York City, without any interactive elementsHere’s the SQL query I used to generate the data for the above panel:SELECT
  max(time) as ""time"",
  vid AS ""vehicle_id"",
  route_id,
  CASE WHEN route_id LIKE 'M%' THEN 1
       WHEN route_id LIKE 'B%' THEN 2 
       WHEN route_id LIKE 'Q%' THEN 3 
       WHEN route_id LIKE 'S%' THEN 4 
       ELSE 0
   END AS ""color"",
  ST_X(geom) AS ""longitude"",
  ST_Y(geom) AS ""latitude""
FROM mta WHERE time > now()-interval '1.5 minutes' 
GROUP BY vid, route_id, geom 
ORDER BY 1;In this query, I use the variablecolorto distinguish between different types of buses based on their route. There are 4 types of bus routes: M, B, S, and Q - corresponding to New York’s boroughs of Manhattan, Brox and Brooklyn, Staten Island and Queens.Then, we use Grafana’s threshold settings to assign each bus type to a color:Threshold settings to give each bus type a unique colorNotice that there are 5 colors - one for each bus route M,B,Q and S, as well as one for routes that don’t fall into those categories.While the standard static panel tells us the live location of the buses, there’s not much that we can do to interact and explore the data more, apart from zooming in and out.Let’s change that by creating a variable to alter which bus routes we display on the map.Step 1: Create a variable as a queryOur goal here will be to create a variable that controls the type of buses we display in the visual, based on the bus’ route.For simplicity sake, let’s define 4 types of bus routes: M, B, Q, and S.Grafana includes many types of variables, and variables in Grafana function just like variables in programming languages. We define a variable, and then when we reference it, we’re referring to the thing we defined the variable as.To create a new variable, go to your Grafana dashboard settings, navigate to theVariableoption in the side-menu, and then click theAdd variablebutton.In this case, we use theQuerytype, where our variable will be defined as the result of an SQL query.How to create a variable of type QueryUnder theGeneralsection, we name our variableroute. Then, we assign it the label of “MTA Bus Route.”Name, label, and type settings for ourroutevariableLabelsin Grafana are the human readable descriptors that appear next to your dashboard’s drop down (picker) menu (see below image).Your variableLabelsdisplay in your final dashboard UI, whilenameis what you use to reference the variable in your queriesSecondly, underQuery options, we define the query that will define the variable.Settings used to create ourroutevariableHere, we select our variable’s data source, which is the database that the query will execute against. In this case, we use “MTA Bus DB”, the PostgreSQL database that houses ourMTA Bus data.Now, we define a SQL query whose results will define ourroutevariableSELECT * from (values ('M'),('B'),('Q'),('S')) v;This query returns the letters M, B, Q, S,  which are the types of buses, based on their route_id (for a refresher, see the SQL query in the prerequisites section).We could also use a query that’s more advanced, but gives the additional benefit of human readable names rather than using symbols or acronyms:SELECT k AS ""__text"", v AS ""__value"" from (values ('Manhattan','M'),('Bronx/Brooklyn' ,'B'),('Queens','Q'),('Staten Island','S')) v(k,v);Here, we define four key value pairs to be the set of possible values for ourroutevariable, where thekey is the human readable name of the bus type and the value is the letter corresponding to the route type.As an aside: This variable could also have been of typeCustom, as we have a static list of values that never changes, allowing us to specify the values directly without the need to specify them through SQL.However, if you want the mapping from key to values, in order to express both symbols and human readable names, a SQL query is required. I’ve used a SQL query in the example, since in practice you often want variables to take on values that aren't hard-coded but that change based on data in the database, such as customer names, cluster names, etc.Next, let’s define how we select our variable.Since we want to see many different types of buses, we enable the multiple selections option, as it’s reasonable to want to see many different types of buses at once.We also want an “All” option to quickly select all the bus types, rather than selecting them one by one.Sometimes you might not want to select multiple options – like if you're selecting metrics from different databases.But, in this case, I find it valuable to see data from multiple bus routes at the same time.To see a preview of the resulting options, we scroll toPreview of Values. In our case, we have  ‘All’, ‘M’, ‘B’, ‘Q’, ‘S’, which are the options we want (i.e., our 4 individual bus routes and the “select all” option).Grafana renders a handy preview of what values of ""route"" will be down in the drop down menuStep 2: Modify your graph query to use your new variableIn this step, we modify our query to use the variable we created in Step 1.For a SQL query, we do this by modifying theWHERE clauseto filter out undesirable results. In our case, we want to show only the bus types that are selected through the drop down menu picker in the UI.Here’s the modified query:SELECT
  max(time) as ""time"",
  vid AS ""vehicle_id"",
  route_id,
  CASE WHEN route_id LIKE 'M%' THEN 1
       WHEN route_id LIKE 'B%' THEN 2 
       WHEN route_id LIKE 'Q%' THEN 3 
       WHEN route_id LIKE 'S%' THEN 4 
       ELSE 0
   END AS ""color"",
  ST_X(geom) AS ""longitude"",
  ST_Y(geom) AS ""latitude""
FROM mta WHERE time > now()-interval '1.5 minutes' 
AND substring(route_id,1,1) IN ($route)
GROUP BY vid, route_id, geom 
ORDER BY 1;The relevant part of the query, where we reference our new variable by the name we defined in Step 1, is this line:WHERE time > now()-interval '1.5 minutes' 
AND substring(route_id,1,1) IN ($route)This line says that our visualization displays only if the first letter of itsroute_idis in the set of allowed routes (as selected by the user through the drop down picker, defined by theroutevariable). The values selected in the picker will define what our$routevariable will be.For example:If a user selects all routes thenroute = (M, B, S, Q), the set of all the types possible.But, if a user selects only the M and B routes, thenroute = (M,B)and we automatically filter out S and Q buses and display only buses with route_id starting with M and B.Once we’ve modified the query, we save our changes and check if our variable works as expected.To do this, we select various routes in the picker and verify that only our selections appear, like so:Our new interactive dashboard, where our map updates based on the routes we select in our dropdown menu.As you can see, our map automatically changes the bus types we see on our NYC map, based on the selection we make in the drop-down! You can also see the change takes place in multiple panels since they all use the variable, $route in their query.That’s it - we’ve successfully created an interactive Grafana visual using variables 🎉.Learn MoreFound this tutorial useful? Here are two more resources to help you build Grafana dashboards like a pro:About TimescaleIf you are not using TimescaleDB yet,take a look. It's a PostgreSQL extension thatwill make your queries fasterviaautomatic partitioning, query planner enhancements, improved materialized views,columnar compression, and much more.If you're running your PostgreSQL database in your own hardware,you can simply add the TimescaleDB extension. If you prefer to try Timescale in AWS,create a free account on our platform. It only takes a couple seconds, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grafana-variables-101/
2020-03-16T17:36:07.000Z,[New Webinar]: How to analyze your Prometheus data in SQL: 3 queries you need to know,"Join us on March 25th to learn how to build the ultimate long term store for Prometheus metrics - complete with demos and queries you can use to analyzeyourmonitoring dataOver the last year, we've met several developers at events like PromCon EU, All Things Open, AWS re:Invent, and most recently demoed all things Prometheus + Grafana + TimescaleDB atdevopsdays NYC. At these events, we’re continually meeting developers looking for a better way to store, analyze, and visualize their monitoring metrics.In my own experience, I’ve run into issues with traditional monitoring setups, be it that they’re too rigid to give me the more custom insights I’m looking for or too cost-prohibitive for me to use in my projects. I heard the same from the developers and community members I met at devopsdays NYC, and I see it again and again on social media andTimescale Community Slack.The good news is that there’s a better way!In my upcoming webinar, I’ll show you how to use open-source software to “roll your own” monitoring solution, allowing you to keep your Prometheus metrics around forever (almost), never run out of disk space, and use SQL to write custom queries.Join me on March 25th at 10am PT/1pm ET/4pm GMTas I use the scenario of monitoring a database to:Demo using Timescale and PostgreSQL to store and analyze your monitoring dataSpin up Grafana dashboards to visualize trendsQuick example of the dashboards we’ll build to visualize our monitoring data and queries.We’ll start with a quick architecture overview and why it’s important to have a long-term data store for your metrics, then get straight into the code and ways you can customize and add metrics that give you more insight (e.g., query latency).During the session, you’ll:Learn why PostgreSQL + Timescale is the ultimate long term store for Prometheus metrics (and why you need a long-term store in the first place)Set up aggregates to rollup hourly and daily summaries of your metricsCreate automated downsampling rules to keep aggregated metrics around longer, without wasting disk spaceSee 3 common monitoring queries that you can use and customize right awayRSVPhere.As always, myself and other Timescale team experts will be available to answer questions throughout the session, and share ample resources and technical documentation.Signup even if you’re unable to attend live, and I’ll make sure you receive the recording, slides, and resources - and answer any questions you may have along the way.See you soon!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/new-webinar-how-to-analyze-your-prometheus-data-in-sql-3-queries-you-need-to-know/
2020-06-11T15:50:40.000Z,"[New Webinar] Grafana 101 Part III: Interactivity, Templating and Sharing","Take your dashboards to the next level with step-by-step demos, queries, and pro tips that’ll get you creating interactive, reusable, and shareable Grafana dashboards in less time.In our Guide to Grafana series, we’ve tackledhow to create awesome visualizationslike graphs, world maps, and gauges, andhow to set up alertsthat notify us about anomalies and when things go wrong in the metrics we care about.The next step is to create dashboards that allow us and our teammates to drill into specific details, get the info we need quickly, and use templated visualizations and queries, instead of starting from scratch every time.InGuide to Grafana 101 Part III: Getting Started with Interactivity, Templating and Sharing, I’ll take you from zero to hero in using Grafana variables and templates to create more interactive and insightful dashboards for your stakeholders – and how to save those visualizations for others (or you) to use in other dashboards.RSVPto join me on June 17th at 10am PT / 1pm ET / 4pm GMT.Can’t make that time? I’ll send the recording and resources to all registrants, so RSVP anyway.Dashboards aren’t just for those who build themThose of us that work with data often want to make useful dashboards that make it easier for ourselves, our team, and our organization to make sense of and get insight about the data we collect.We touched on this in “Grafana 101: How to Build (awesome) Visualizations,” but two common problems I’ve run into, both when creating dashboards and using them as a stakeholder:Stakeholders who aren’t familiar with the underlying system architecture are unable to interact or filter visualizations, which leaves them unable to answer their questions.Engineers get stakeholder requests to write new code or change the underlying queries powering the dashboard.Moreover, building dashboarding for complex systems with many components, like apps composed of microservices, is time-consuming and involves a lot of repetition, such as creating graphs for the same metric on different servers.In my upcoming Guide to Grafana session, I’ll show you how to use Grafana variables and templates to overcome these issues. You’ll learn how to build more interactive and insightful dashboards for your stakeholders, while also reducing repetitive work.In summary: better dashboards, in less time.What you'll learnAs always, I’ll focus on code and step-by-step live demos.We’ll use two scenarios: monitoring an app, composed of microservices and running in Kubernetes (multiple servers where we want to visualize the same metric) and monitoring the live location of buses in New York City (where we want to share dashboards with teammates).I’ll take you through creating variables and using them in your dashboards to make them more interactive, as well as saving templates for canonical queries and visualizations to reuse in other dashboards and share with your organization.More specifically, you’ll:Get a quick overview of what, when, and why to template your dashboardsLearn the difference between a template and a variableCreate different types of variables, like query and customCreate dynamic dashboards that will save you time when you’re dashboarding large projectsSee template examples for DevOps and IoT Monitoring datasetsSet up version control and share your dashboards with teammatesHear pro tips and recommendations to customize and apply what you learn to your own projectsSneak peak of creating an interactive dashboard to monitor IoT devicesMy goal is that you leave the session with an understanding of when, why, and how to use variables and templates in your Grafana dashboards, as well as the resources you need to integrate them into your own setup.Myself and other Timescale Grafana experts will be available to answer questions and share resources throughout the session, so you leave ready to continue mastering Grafana.I hope to see you there – but, as I mentioned,sign up even if you’re unable to attend live. I'll send you the resources and recording and happily answer any questions 1:1.Want to explore Grafana in the meantime?Check out our in-depth tutorials and quickstarts.See you soon!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/new-webinar-grafana-101-part-iii-interactivity-templating-and-sharing/
2020-03-13T16:04:03.000Z,"Devopsdays NYC 2020 demo, open space recap & more","Learn about the latest devopsdays event, get our demo, answers to community questions, and more.We recently attended the NYC installment of thedevopsdays event series(thank you to the local organizers and volunteers!), where we met with community members interested in all things monitoring, infrastructure, software development, and CI/CD.Given the cancellation of many industry events to ensure public safety and mitigate COVID-19’s spread (check out our blog post if you’re interested in monitoring it yourself), we’re sharing a bit about our recent experience – what we learned, what we demoed, and what we spoke about – to bring the event experience to the wider community.The DemoMatvey and Avthar ready to demo and meet community members @ Devopsdays NYCDuring the event, I demoed how to use TimescaleDB as a long-term store for Prometheus metrics - combining Prometheus, TimescaleDB, and Grafana to monitor a piece of critical infrastructure (in this case, a database). This sort of create-your-own flexibility and customization is becoming more and more common in the conversations I have with developers, and this demo allows you to create a monitoring stack that suits your needs, without adding significant costs.Why this scenario? I was inspired by one of our Managed Service for TimescaleDB customers, who uses TimescaleDB to store and analyze their Prometheus metrics. They told us how it not only saves them money and disk space, but it also allows them to keep their data around and see trends over longer time periods.See the demo in actionHow (and why) to use TimescaleDB as a long-term store for your Prometheus metricsYou’ll notice a Grafana dashboard visualizing metrics, with TimescaleDB as the data source powering the dashboard. I focused on the below basic monitoring metrics, but if you try it yourself, you can customize and add more metrics that give you more insight (e.g., query latency, queries per second, open locks, cache hits, etc.):CPU usageService status% of Disk used# of Database connections% Memory usedNetwork StatusTo replicate the demo, follow these tutorials onhow to store Prometheus metrics in Timescaleandhow to use Timescale as a datasource to power Grafana dashboards.Open Space: DevOps & DataMatvey Ayre leading the Open Space discussion on DevOps DataDevopsdays “Open Spaces” are a (wonderful) conceptsimilar to an unconference format: there’s a block of time scheduled for any attendees to discuss topics of their choosing with other interested attendees. Simply propose a topic to the audience that you’d like to discuss for 30 mins and other attendees can pick and choose which sessions they’d like to attend.Fellow TimescalerMatvey Aryeand I hosted an Open Space session about DevOps Data, and other topics ranged from negotiating pay and other soft skills to DevOps in small companies and DevOps in a certain ecosystem (AWS, Microsoft Azure, Google Cloud, etc.).In our session, we heard stories, best practices, and the ways developers from all industries and areas think about the DevOps data they collect.A few highlights and commonalitiesTeams are moving away from managing infrastructure themselves and toward managed services(as one person put it: “One of the key criteria when we select a new tool is that we want one less thing to manage”).DevOps at certain companies can be a lonely and isolating job. To remedy that, folks mentioned that they’d joined (and recommend!) a few Slack workspaces:O11y.slack.com,HangOpsandCoffee Ops.Data is becoming increasingly central in how teams fuel their post-mortem problem analysis. Developers collect data about critical incidents, search for patterns in what’s causing them, and correlate this information with how it impacts clients or users.One team’s best practice and advice (they manage a massive consumer messaging app): Take snapshots of high load periods. This way, you get more detailed information to use for planning and to calibrate for the following years. In this team’s case, the New Year’s Eve timeframe is when they see the highest number of messages sent across their global user base.Kubernetes, as always, was a hot topic. Two common pain points stood out (and are things that we can relate to as webuild our Kubernetes deployment and multi-node offerings):#1: Visibility about what’s happening inside clusters and pods. Someone summed it up with, “I don’t just want to know my pod is offline, I want to know what was going on inside it.” We couldn’t agree more.#2: Aggregate observability data across clusters to simplify things for Ops teams who handle metrics from multiple applications teams.Questions & ConversationsTo me, the best part of any conference are the hallway conversations and hearing the things community members are keen to learn. As a company, we’re help-first, so, in the spirit of helping, here are a few questions I heard again and again that may be relevant as you get up and running, or do more advanced things with TimescaleDB:How does TimescaleDB perform at scale?TimescaleDB scales up well within a single node, and also offers scale-out capabilities if you use ourmulti-node beta.In our internal benchmarks on standard cloud VMs, we regularly test TimescaleDB to 10+ billion rows, while sustaining insert rates of 100-200k rows per second (1-2 million metric inserts / second). While running on more powerful hardware, we’ve seen users scale a single-node setup to 500 billion rows of data, while sustaining 400k row inserts per second. To learn more about how TimescaleDB is architected to achieve this scale, see thisblog explainer.And, in our internal tests, a multi-node beta setup with 9 nodes achieved an insert rate of over12 million metrics per second(and you can read more about our multi-node benchmarkinghere).What’s the role of a long-term data store? What types of things does this allow me to do?In order to keep Prometheus simple and easy to operate, its creators intentionally left out some of the scaling features developers typically need. Prometheus stores data locally within the instance and is not replicated. While having both compute and data storage on one node makes it easier to operate, it also makes it harder to scale and ensure high availability.More specifically, this means Prometheus data isn’t arbitrarily scalable or durable in the face of disk or node outages.Simply put, Prometheus isn’tdesignedto be a long-term metrics store. However, its creators also made Prometheus extremely extensible, and, thus, you can use TimescaleDB to store metrics for longer periods of time, which helps with capacity planning and system calibration. This combination also enableshigh availabilityand providesadvanced capabilities and features, such as full SQL, joins and replication (things not available in Prometheus). To learn more, seewhy use TimescaleDB and Prometheus.How do I use TimescaleDB and Prometheus? Do I have to use any special connectors?Check out the demo :). I suggest using TimescaleDB as a remote read and write for Prometheus metrics, whether they’re infrastructure for an internal system or your public-facing eCommerce website. Since TimescaleDB extends Postgres, you use thepg_prometheus extensionfor Postgres and ourprometheus_postgresql_adapter,and you’re ready to get started.Whatever works with Postgres works with TimescaleDB, so, if you want to connect toviz tools(likeGrafanaorTableau), ingest data from places likeKafkaor insert and analyze data using your favorite programming language (like Python or Go), just use one of the many connectors and libraries in the Postgres ecosystem.Wrapping UpThank you again to the devopsdays NYC team for your work to pull off such an interactive, fun, and community-first event! We’ll definitely be attending as future events are announced (virtually or otherwise).In the meantime, those resources once more:Demo VideoTutorials:Prometheus,Grafana...and, in the event you’d like to see an advanced version of this demo and/or are keen to join some #remote-friendly events,you can join me on March 25 for “How to Analyze Your Prometheus Data in SQL: 3 Queries You Need to Know.”I’ll focus on code and showing vs. telling: You’ll learn how to write custom SQL queries to analyze infrastructure monitoring metrics and create Grafana visualizations to see trends, and I’ll answer any questions that you may have.Interested?Sign up here. You’ll receive the recording and resources shortly following the session, soregistereven if you can’t attend live.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/devopsdays-nyc-2020-demo-open-space-recap-more/
2019-09-25T16:52:32.000Z,How to Quickly Build SQL Dashboards for Time-Series Data,"Achieve automated materialized views using TimescaleDB continuous aggregates.Time-series dataprovides significant value to organizations because it enables them to analyze important real-time and historical metrics. However, data is valuable only if it’s easy to access. That’s where being able to build SQL dashboards that run repetitive analytical queries becomes a force multiplier for organizations looking to expose their time-series data across teams.In this post, we will discuss why you should use TimescaleDBcontinuous aggregatesto speed up queries that aggregate over time. Additionally, we will walk through asample applicationthat leverages this feature specific to TimescaleDB.Why continuous aggregates?Powering real-time dashboards is one of the top use cases for TimescaleDB. Dashboards are unique in that they are often pre-built and run repetitive queries. However, any database powering dashboards also needs to be able to support ad-hoc queries since this increases the agility of creating, updating, and adjusting dashboards.PostgreSQL excels at these ad-hoc queries by offering the flexible SQL interface. However, as data sizes increase, PostgreSQL tends to fall over. That’s where TimescaleDB comes in!With automated partitioning, TimescaleDB can optimize analytical time-series queries so that less data is read from disk. We’ve also built continuous aggregates, which further reduce disk throughput and compute requirements when running historical aggregate queries. By combining the inherent flexibility of SQL with the advanced capabilities of TimescaleDB, you can perform the ad-hoc queries needed to define your dashboards, as well as efficiently run repetitive analytical queries to power ongoing live dashboards.Should I be using continuous aggregates?We built continuous aggregates to significantly speed up repetitive queries that calculate aggregates over periods of time. It is not really meant for point queries, e.g. queries that search for values at a single or short period of time. Essentially, a continuous aggregates populates a materialized view that stores aggregates on a scheduled basis. By storing aggregates, it minimizes the computation required to run an aggregate query and the amount of data that needs to be read off disk.TimescaleDB continuous aggregates do have some specific characteristics that you should be aware of as you decide if you want to use this feature.Continuous aggregates are scheduled.You define how up-to-date you want your aggregates to be. If you want the aggregates to be updated at the same time a value is written into the original hypertable, you’ll end up incurring write amplification at the time of insert. This will result in lower inserts per second. We typically recommend users looking for fast insert performance to allow aggregates to lag behind the bleeding insert edge.Continuous aggregates support out-of-order inserts.Some systems that implement continuous aggregates actually ignore out of order inserts. In TimescaleDB, when the scheduled job runs to update the continuous aggregate, we also check for any out of order inserts and update those associated aggregates.The continuous aggregate itself is actually a hypertable.You don’t have to worry about how to scale that particular table.Walking through a sample application leveraging continuous aggregatesLet’s walk through how continuous aggregates work be exploring a sample application. We first introduced this application in ourtime_bucket() analytics post, where we showed how you could build flexible graphs using TimescaleDB’stime_bucket()function.The sample application is written in Python. It essentially scrapes the Open AQ (air quality) API, parses the results, and stores all measurements collected from air quality sensors for all cities in Great Britain. You can check out the codehere.The Grafana dashboard locatedheredescribes step-by-step how to set up multiple continuous aggregates. To give you a quick sense of why continuous aggregates are really useful, here’s a query that we use in the Air Quality example:SELECT
  time_bucket('1 day', time) as bucket,
  parameter_id,
  avg(value) as avg,
  max(value) as max,
  min(value) as min
FROM
  measurements
GROUP BY bucket, parameter_id;You’ll notice that we are querying across all time and bucketing things by 1 day intervals. The computations of bucketing things by 1 day, as well as reading all that data off disk is high. The table at the point at which I’m running this query has ~700k rows. You can imagine how this query could get really slow as you add more data.Below is an example query plan. Notice how much work is required to compute this query!To speed this up, I wrote a continuous aggregate.Now, let’s try to query the same thing, but query the measurements_daily table directly. The query plan is greatly simplified in that it is no longer scanning as much data.Advanced configurations and togglesWhen we designed continuous aggregates, we wanted users to be able to toggle and configure the characteristics that they wanted to achieve from continuous aggregates. You most likely won’t need to touch a lot of the toggles, but here are a couple that you should definitely be aware of.timescaledb.refresh_interval:This configuration specifies how often you want a continuous aggregate to run. Note that when a continuous aggregate is run, it only runs on new data that it hasn’t observed before. This includes any data that came in out of order that invalidates older continuous aggregates. The value you choose here impacts how hard your background workers are working in order to calculate continuous aggregates.timescaledb.refresh_lag:This configuration specifies how far behind a continuous aggregate runs compared to the insert bleeding edge. This impacts write amplification. If you set the refresh_lag to the negative bucket width, the continuous aggregate will immediately calculate and update the underlying materialized view whenever new data is received. However, this also means that you are executing multiple writes on insert, which impacts insert performance.timescaledb.max_interval_per_job:Some of our users using continuous aggregates for the first time notice that their aggregates never populate. This is a situation where you want to take a look at this value - it controls how long a query can run in the background before we kill it. If you take a huge table and try to create a continuous aggregate on it for the first time, it’s likely that you’ll hit this barrier.The best way to check if your configurations make sense is to usetimescaledb_information.continuous_aggregate_stats. These stats will show you important statistics about your continuous aggregates.Next stepsAs you can see, continuous aggregates can be an extremely useful function when you are looking to further reduce disk throughput and compute requirements when running historical aggregate queries.If you are ready to try it out for yourself, check out thistutorial. If you are brand new to TimescaleDB, follow theinstallation instructionsor get started withManaged Service for TimescaleDB.Need help along the way? You can always reach out to us on our communitySlackchannel.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-quickly-build-dashboards-with-time-series-data/
2019-08-08T15:20:32.000Z,Simplified Time-Series Analytics Using the time_bucket() Function,"What Is Time-Series Analytics?Time-series analytics refers to the process of analyzing and extracting insights from data that is organized and collected over time, typically at regular intervals—or, in other words, time-series data.If you are working with time-series data, you need a way to be able to easily manipulate, query, and visualize that data to perform time-series analytics. What you may or may not already know is thatTimescaleDBprovides a number of time-oriented functions thataren't found in traditional relational databases.These functions are meant to provide two key benefits: improved ease of use for time-series analytics and improved performance. The functions live alongside full SQL and can be viewed as extensions of the SQL language. Since many developers today already know SQL, the learning curve is greatly reduced.Two critical TimescaleDB time-series functions are:time_bucket()andtime_bucket_gapfill(). Time_bucket() is used for aggregating arbitrarily-sized time periods, and gapfill() is important when your time buckets have missing data or gaps, which is a very common occurrence when capturing thousands of time-series readings per second. Together, both of these are essential for analyzing and visualizing time-series data.In this blog, we’ll discuss both of these capabilities and show them in action using Grafana.Time-series analytics: Background ontime_bucket()Essentially,time_bucket()is a more powerful version of the standard PostgreSQLdate_trunc()function.date_trunc“truncates” a TIMESTAMP or an INTERVAL value based on a specified date part (e.g., hour, week, or month) and returns the truncated timestamp or interval.For example,date_trunccan aggregate by one second, one hour, one day, or one week. However, users often want to see aggregates by five minutes or four hours, etc. This can get pretty complicated in SQL, buttime_bucket()makes it easy.Time bucketing allows for arbitrary time intervals (e.g., five minutes, six hours, etc.), as well as flexible groupings and offsets, instead of just second, minute, hour, and so on.In addition to allowing more flexible time-series queries,time_bucket()also allows you to write these queries in a simpler way. In fact, it can infer the time range directly from the WHERE clause, which greatly simplifies the query syntax.Here it is in action:SELECT time_bucket('5 minutes', time) AS five_min, avg(cpu)
  FROM metrics
  GROUP BY five_min
  ORDER BY five_min DESC LIMIT 12;By the way, this time-series analytics feature has actually been a core function of TimescaleDB since its first release in April 2017, and our users love it, particularly for time-series analytics!When to Usetime_bucket()for Time-Series AnalyticsAs you can imagine, time bucketing can be helpful for a number of scenarios. When it comes to creating dashboards or visualizations of time-series data, many rely on this function to turn their raw observations into fixed time intervals.When graphing time-series data using a solution such asGrafana, aggregations can help identify trends over time by grouping raw data into higher-level aggregates. For example, you might want to average monthly raw data daily to achieve a smoother trend line or count the number of occurrences of non-numeric data.Building on top of time_bucket() with time_bucket_gapfill()One issue users often encounter when working with time-series data is recording measurements at irregular or mismatched intervals. This creates a time bucket interval with missing data or gaps.Fortunately, TimescaleDB has a function calledtime_bucket_gapfill()that allows you to aggregate your data into continuous time intervals. You can choose two different ways to fill in the gaps:locf()which carries the last known value in the time range forward orinterpolate()which does a linear interpolation between gaps.Time_bucket_gapfillwas introduced in TimescaleDB 1.2 as a Community feature. To learn more about gapfill, check out thisblog post.Get Hands-On With a Sample ApplicationNow that you know whattime_bucket()andtime_bucket_gapfill()do, it’s time to get hands-on with a sample application for time-series analytics!We’ve built a simple Python application that pulls data from the Open AQ Platform API (an open API air quality data). Essentially, the application reads the API to get air quality values for all cities in Great Britain over a period of time, parses the results, and stores all measurements collected from air quality sensors in TimescaleDB.All the code for this application can be found on GitHubhttps://github.com/timescale/examples/tree/master/air-quality.We also connected a Grafana instance to the TimescaleDB database that stores these results and wrote a step-by-step tutorial on how to usetime_bucket()andtime_bucket_gapfill()to visualize the data.You can build your own dashboard usingTimescale(which runs both TimescaleDB and Grafana)!Next StepsThetime_bucket()function is just one example of how TimescaleDB offers optimized SQL functions for working with time-series data and time-series analytics. To learn more about other TimescaleDB functions, check out ourdocs.If you’re ready to get started, you candownload TimescaleDBor sign up forTimescaleto quickly get Timescale and Grafana instances up and running (free 30-day trial, no credit card required).Like this post and are interested in learning more? Sign up for our mailing list below (right-hand side of the screen), or follow us onTwitter.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/simplified-time-series-analytics-using-the-time_bucket-function/
2018-07-12T15:00:00.000Z,Uniting SQL and NoSQL for monitoring: Why PostgreSQL is the ultimate data store for Prometheus,"How to use Prometheus, PostgreSQL + TimescaleDB, and Grafana for storing, analyzing, and visualizing metrics.Jump to:Tutorial on how to use Prometheus + PostgreSQL + TimescaleDB.If you ever needed a monitoring solution for your infrastructure then you’ve probably heard aboutPrometheus, an open-source community-driven monitoring system that also includes metrics collection and alerting.In fact, the rise ofKuberneteshas made Prometheus even more popular. If you look at their shared history, it’s easy to understand why. Google developed and open-sourced Kubernetes based on their decade-long experience with their own cluster scheduling system calledBorg. Prometheus was initially developed at SoundCloud and heavily inspired by Borgmon, the internal monitoring system Google developed for Borg. In a way, Kubernetes and Prometheus originate from the same lineage, and are a perfect fit for each other. They also both happen to be 10-letter Greek words. (More on the history of Prometheus here.)Prometheus is quite simple to start with. What makes Prometheus awesome is its unapologetic approach to solving monitoring in a simple and straightforward way. Its philosophy is to do one thing, and do it well. You can see this approach reflected in thePromQLlanguage which is known for being powerful, yet simplistic.However, Prometheus’ philosophy can also be limiting. To their credit, the developers of Prometheus foresaw that their product is opinionated, and built in extensibility to allow other systems to improve on it. In turn, Prometheus users often look to other systems as a way to augment their monitoring setup. This is where PostgreSQL comes in.In particular, Prometheus users often turn to PostgreSQL for the following reasons:Scalable, durable, long-term data storageOperational ease with a broad tooling ecosystemSQL query power, flexibility, and “future-proofing”Debunking monitoring misconceptionsThis is where we typically hear a few objections:I thought PostgreSQL doesn’t scale for metrics?I thought PostgreSQL required schemas (and I don’t want to worry about schemas)?I thought PostgreSQL didn’t connect to Grafana?I thought SQL doesn’t work for analyzing metrics data?In this post we’ll discuss why these are actually misconceptions, and how new tools (many developed by TimescaleDB engineers) overcome these objections. But at a quick glance:From this list, the biggest misconception we’ve encountered is that PostgreSQL can’t scale for metrics. This is a fallacy that has already been disproven byTimescaleDB.TimescaleDB improves on PostgreSQL in a variety of ways:20x higher inserts, 2000x faster deletes, 1.2x-14,000x faster queries. TimescaleDB also introducesnew functions(e.g.,time_bucket()for aggregating by arbitrarily-sized time periods) that are necessary for metrics analysis. It does this while still looking and feeling like PostgreSQL, e.g., by still supporting all PostgreSQL commands and full SQL.(Note that TimescaleDB is packaged as a PostgreSQL extension, not a fork. For more information:How TimescaleDB works.)Now, let’s get back to why PostgreSQL (andTimescaleDB) is the ideal long-term store for Prometheus.Scalable, durable, long-term data storageIn order to keep Prometheus simple and easy to operate, its creators intentionally left out some of the scaling features one would normally expect. The data in Prometheus is stored locally within the instance and is not replicated. Having both compute and data storage on one node may make it easier to operate, but also makes it harder to scale and ensure high availability.As a result, Prometheus is not designed to be a long-term metrics store. From the Prometheusdocumentation:[Prometheus] is not arbitrarily scalable or durable in the face of disk or node outages and should thus be treated as more of an ephemeral sliding window of recent data.This means that if you want to store your Prometheus data in a scalable, durable way for more analysis, you will need to complement Prometheus with a time-series database (since metrics data is time-series data).Traditionally, PostgreSQL has had scalability challenges that have prevented its use as a metrics store. However, thanks to TimescaleDB, PostgreSQL can now easily handle terabytes of data in a single table (as we discussed earlier).PostgreSQL also supportshigh availability and replication, further making it a good fit for long term data storage. In addition, it provides advanced capabilities and features, such as full SQL, joins, even geospatial support viaPostGIS, which is simply not available in Prometheus.So, how does PostgreSQL integrate with Prometheus to store data? All the metrics that are recorded into Prometheus are first written to the local node and then written to PostgreSQL (more information below). This means that all of your metrics are immediately backed up, so that any disk failure on a Prometheus node will be less painful.PostgreSQL can also store other types of data (metrics, relational, JSON), or even other sources of time-series data (via TimescaleDB) allowing you to centralize your monitoring data from different sources and simplify your stack. You can even join those different types of data and add context to your monitoring data for richer analysis.Since one needs TimescaleDB to store and properly analyze metrics data in PostgreSQL, from here on we’ll talk about TimescaleDB, with the full understanding that TimescaleDB looks and feels just like PostgreSQL.Operational ease with a broad tooling ecosystemThere are severalstorage optionsfor Prometheus. However, the challenge with all of them is that they will likely require your team to operate and manage yet another system.TimescaleDB is an exception. It operates just like PostgreSQL, which means that teams who already have PostgreSQL experience can re-use that knowledge for their TimescaleDB Prometheus storage. TimescaleDB also inherits the broad PostgreSQL ecosystem of tooling, management, connector, and visualization options like Kafka, Apache Spark, R/Python/Java/etc, ORMs, Tableau, etc.In particular, given the decades of development invested in streamlining PostgreSQL, it (and TimescaleDB) is often (and perhaps surprisingly) more resource efficient than other newer data stores that were specifically built from the ground up for metrics. As one Prometheus user recently told us:Being able to use the lighter-weight and scalability features of PostgreSQL are the big thing. In comparison, another NoSQL store we tried was a memory hog. The future of being able to do more granular queries is great, too.Other operational improvements relate to how Prometheus works. The only way to scale Prometheus is byfederation. However, there are cases where federation is not a good fit: for example, when copying large amounts of data from multiple Prometheus instances to be handled by a single machine. This can result in poor performance, decreased reliability (an additional point of failure), duplicated data, or even loss of data. These are all the problems you can “outsource” to TimescaleDB.SQL query power, flexibility, and “future-proofing”PromQL is the Prometheus native query language. It’s a very powerful and expressive query language that allows you to easily slice and dice metrics data and apply a variety of monitoring-specific functions.For example, here is a query that is perhaps better expressed in PromQL than SQL:Max CPU frequency by CPU core in the last 5 minutes:PromQL:max(max_over_time(node_cpu_frequency_hertz[5m])) by (cpu)SQL:SELECT labels->>'cpu', MAX(value)
FROM metrics
WHERE name='node_cpu_frequency_hertz'
      AND time > NOW() - interval '5 min'
GROUP BY labels->>'cpu';However, there may be times where you need greater query flexibility and power than what PromQL provides. For example, when:Trying to cross-correlate metrics with events and incidents that occurredJoining metrics data with other data sources (e.g., infrastructure inventory or weather data)Running more granular queries for active troubleshootingConnecting to other visualization tools like TableauConnecting to Python/R or Apache Spark to apply machine learning or other forms of deeper analysis on metricsFor example, here is a query that is more naturally expressed in SQL:Check whether system performance was impacted by a Linux kernel updatePromQL: Hard to do.In PromQL, this would be hard to do, unless this information (e.g., live kernel update history) was already stored in Prometheus. Considering the problems Prometheus has withhigh-cardinality datasets, this would require storing data in a normalized way. Even then, adding external metadata to Prometheus can be cumbersome, while in SQL it would be just an insert. Also, in many cases, storing external metadata in Prometheus may not be feasible, or in the case of certain types of data (e.g., weather, user, business) may not even make sense.SQL:SELECT time_bucket('1 hour', m.time) AS hour_bucket, 
       m.labels->>'host', h.kernel_updated, AVG(value)
FROM metrics m LEFT JOIN hosts h on h.host = m.labels->>'host' 
AND  time_bucket('1 hour', m.time) = time_bucket('1 hour', h.kernel_updated) 
WHERE m.name='node_load5' AND m.time > NOW() - interval '7 days'
GROUP BY hour_bucket, m.labels->>'host', h.kernel_updated
ORDER BY hour_bucket;In this case, we are using an additional table containing information about Linux kernel updates per host, and then easily joining that data with metrics from Prometheus to figure out if maybe some kernel patch decreases system performances.Sometimes you just want to “future-proof” your system such that it will support not only the queries you want to run today, but also what you’d like to run in the future.This is where the full SQL support of TimescaleDB can be of great help, allowing you to apply the full breadth of SQL to your Prometheus data, joining your metrics data with any other data you might have, connect to other data analysis and visualization tools, and run more powerful queries. (Some more exampleshere.)Prometheus + TimescaleDB + PostgreSQL = Better togetherBy using Prometheus and TimescaleDB together, you can combine the simplicity of Prometheus with the reliability, power, flexibility, and scalability of TimescaleDB, and pick the approach that makes most sense for the task at hand. In particular, it is because Prometheus and TimescaleDB are so different that they become the perfect match, with each complementing the other. For example, as mentioned earlier, you can use either PromQL or full SQL for your queries, or both.To make this work there are two components (both developed by TimescaleDB):1. ThePrometheus PostgreSQL Adapter2. ATimescaleDBdatabase withpg_prometheusThe adapter is basically a translation proxy that Prometheus uses for reading and writing data into PostgreSQL/TimescaleDB. And the adapter has a dependency on the pg_prometheus PostgreSQL extension, which takes care of writing the data in most optimal format for storage and querying within PostgreSQL/TimescaleDB.In particular, pg_prometheus implements the Prometheus data model for PostgreSQL. That means you can write Prometheus data directly to PostgreSQL/TimescaleDB using the familiarexposition format, even without Prometheus. (In other words, it ensures that you don’t have to worry about schemas.)We’ve actually created a step-by-step tutorial to walk Prometheus users through this processes which you can findhere.Bonus! Using the TimescaleDB data source with GrafanaWe also know thatGrafanais another popular platform for analytics and monitoring. But until recently, databases supported by Grafana have been NoSQL systems, offering restrictive SQL-like or custom query languages that were limited in scope, and designed for specific data model and architectures in mind. Not anymore.Timescale engineerSven Klemmbuilt a PostgreSQL data source to use with Grafana to query TimescaleDB directly to visualize your Prometheus data. This is often recommended for users who prefer familiarity and/or an easier query language. In this case, byconnecting Grafana to Prometheus, you still get access to TimescaleDB indirectly through Prometheus as the long-term store of your data. Using this method, Prometheus works more like a cache. If it doesn’t already have the requested information cached, Prometheus will fetch the data from TimescaleDB, cache it, and return the result.Yet soon we will be able to do more. We’re in the process of building a better query editor for TimescaleDB in Grafana, to make it much easier to explore the available time-series data and generate more of these queries automatically.The TimescaleDB query builder for Grafana will look something like this:Preview of the upcoming TimescaleDB query builder for Grafana (full video).Stay tuned for our step-by-step tutorial on how to use the Grafana query editor. But for now, we encourage you to take the time to read the tutorial onGetting started with Prometheus, PostgreSQL, and TimescaleDB.(Update: VisitGrafana’s blogto learn how to make time-series exploration easier with the PostgreSQL/TimescaleDB query editor.)Next stepsFollow the tutorial:Getting started with Prometheus, PostgreSQL, and TimescaleDBRead more of our blog posts:High availability in PostgreSQL,How to scale PostgreSQL 10 using table inheritance and declarative partitioningRead our benchmarks: TimescaleDB vs.PostgreSQL,PostgreSQL 10,Cassandra,MongoDBLike this post? Please recommend and/or share.Want to learn more? Join ourSlack communityand check out ourGitHub.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/sql-nosql-data-storage-for-prometheus-devops-monitoring-postgresql-timescaledb-time-series-3cde27fd1e07/
2020-04-30T18:26:31.000Z,“Guide to Grafana 101: Creating Awesome Visualizations” Recap and Resources,"Get step-by-step demos and learn how to create 6 different visuals for DevOps, IoT, geospatial, and public data scenarios (and beyond).We just kicked off a new “Grafana 101” webinar series, a set of technical sessions focused on getting you up and running withGrafana’s various capabilities (there are tons!) - complete with step-by-step demos, tips, and resources to re-create what we cover.In the first session, “Grafana 101: Creating Awesome Visualizations,” I show how to build 6 common Grafana visuals, plus a few tips and tricks to help you along the way, so you leave ready to create useful, interactive, and awesome Grafana dashboards for your projects.We know not everyone could make it live, so we’ve published therecordingandslidesfor anyone and everyone to access at any time.If you missed this one, or attended and are keen to learn more, we’re hosting the next installment,“Guide to Grafana 101: Getting Started with Alerts” on May 20th(RSVP here).Missed the session? Don't worry, the recording is right hereWhat you’ll learn:Based on my own experience with Grafana and my conversations with other developers, building useful Grafana visualizations isn’t always easy.So, to reduce that learning curve, I created this session to illustrate the common visualization types, datasets, and tips and tricks (likevariables,series-override,  andthresholds) that have helped me get up and running.I made sure to cater to new and existing Grafana users, quickly covering why you’d use Grafana to build dashboards and a quick overview of how Grafana is structured (terminology, hierarchy, etc.). From there, I go straight into step-by-step demos to show you how to build 6 common visuals, in 3 scenarios: DevOps, IoT monitoring with geospatial data, and public datasets.More specifically, the session consists of 4 parts:Why use Grafana?In my opinion, Grafana is a great choice for dashboarding for three main reasons:It’s open source, which makes it a cheaper alternative to proprietary viz tools.It supports many different visualization typesand also has support for alerting (which happens to be the focus of ournext technical session).It integrates with many common data sources, like PostgreSQL, Prometheus, and AWS CloudWatch. Since Grafana allows you to combine data from different sources in one dashboard, you can build highly customized dashboards for cases like monitoring multiple deployment clusters, or deployments of multiple products.Example of a dashboard with panels powered by different data sourcesOrientation and Data Source SetupIn this section, I introduce a basic “mental model” for thinking about how Grafana is structured, introducing its hierarchy of instances, data sources, dashboards, and panels.You’ll also get a walkthrough of a dashboard powered by multiple data sources and see how to connect a TimescaleDB database to Grafana (I’ll use TimescaleDB as my datasource to demo various visualizations).How to set up TimescaleDB as a datasource in GrafanaLet’s code: 6 (awesome) visualizationsAfter getting oriented with Grafana, it’s time for the best part: creating the visuals!I spend 45 minutes taking you through how to create 6 visuals, using the use cases of DevOps, IoT and geospatial data, and charting the spread of COVID-19.DevOps: Monitoring a production database using Prometheus and TimescaleDBIf you’re working in a DevOps scenario, you likely need to monitor various key metrics, and, in the case of my demo, I want to monitor my production database.Visuals we'll create from the DevOps datasetYou’ll see how to set thresholds and build visualizations to answer questions about database performance and status.Grafana visualization types and questions we answer:Single Stat: Which version of the database are we running?Gauge: How much disk space are we currently using? Is it within an acceptable range?Graph:What’s the cache-hit ratio for my database over time?Single Stat with status: What’s the cache-hit ratio for queries in my database? Is it in an acceptable range?Note: We use the dataset fromthis tutorialon using Timescale as a long term datastore for Prometheusmetrics.And, we’re at work on a new connector for Prometheus. If you’re interested in learning more and trying it out, you can check out ourdesign docandGitHub repo.IoT / GeoSpatial: Monitoring Public Bus Locations in New York CityUsing data from theNew York Metropolitan Transport Authority, I show you how to combine time-series and geospatial data to display the real-time location of various buses throughout New York City.Visualize the real-time location of New York City buses using the Worldmap visualIn this section, I focus on one of my favorite visualization types, the Worldmap Panel.Grafana visualization type and question we answer:Worldmap:where are the public buses in New York City right now? Does a certain route or burrough have more buses on the streets at a given time?As a bonus, I also show you how to usevariablesto allow users to filter the bus routes and locations that appear on the map. This turns the dashboard from a static visual into an interactive one, where anyone can select various routes they want to see from an interactive drop down menu.You can learn more aboutvariablesand how they workhere.Public Data: Charting and analyzing trends in the spread of COVID-19 in the USAFinally, I chart the spread of COVID-19 in the USA, using publicly available data from theNew York Times.Learn how to useseries-overridein this example using COVID-19 dataGrafana visualization type and questions we answer:Graph: How many COVID-19 cases and deaths have taken place in the USA to date? How do these trends compare?You’ll also see how to use Grafana’sseries-overridefunction to plot your data on two Y-axes and more accurately show COVID-19 case and death rates over time (something that’s lost due the variables relative scale when both variables are plotted using the same Y-axis).Resources + Q & AWant to re-create the dashboards shown in the demo? Or perhaps you want to try modifying them to work with your own data sources for your projects? Don’t sweat, we have you covered!In the session, we link to several resources, like tutorials and sample dashboards, to get you well on your way:Import our demo dashboardas a starting point for your own Grafana creations (includes variables, queries, and parameters).Learn how to get up and running with Grafana and TimescaleDB in ourGrafana mega-tutorialorget started visualizing your Prometheus metrics.Follow along with the session recording.Get started with Timescale Cloud(our hosted time-series database, which comes with a 30-day free trial)Join our Slackto ask questions and get help from our engineers and community members.Community questionsWe received over 30 questions during the session (thank you to everyone who submitted one!). Here’s a selection:Q: Is the PostgreSQL datasource ""stock"" in Grafana, or a TimescaleDB-specific one?A:The PostgreSQL data sourceisstock in Grafana and was (fun fact) actually contributed by TimescalerSven Klemm.While you can use the PostgreSQL data source in Grafana for Postgres instances that don’t contain TimescaleDB;if you’re using TimescaleDB,we strongly recommend enabling TimescaleDBwhen setting up your data source, in order to get better performance (like I mention in the demo and as outlined inthis tutorial).Q: The latest PG version in the datasource in the demo was 10. What about later versions of PostgreSQL?A:Grafana supports PG 11 and PG 12, but if you’re using Grafana 6.5.2, you may find that the version picker doesn’t include versions above PG 10 (as noted in thisforum post).From my reading, this has been fixed in the latest Grafana releases (6.6 and 6.7), so you can connect your TimescaleDB instances with PG 11 (andnow PG 12!) to Grafana.Q: When should I use a Gauge vs. a Single Stat?A:Single Stats and Gauges both show a single number.The main difference is that Gauges are generally for measuring the current state of consumption of fixed resources (e.g disk, memory, CPU etc.), or cases where it’s important to know usage relative to thresholds. For example, gauges are ideal to quickly let you know that you are currently at 80% of storage capacity.On the other hand, Single Stats are well-suited for displaying current values (like counts), things like version numbers, or other one-off values.Q: From a performance perspective, is it better to give reporting users pre-built views (like aggregates) or the raw time-series data directly?A:If you’re optimizing for performance with regard to query speed and ingestion rate, using views, like TimescaleDB’scontinuous aggregates, is the way to go.That way, your reporting users query the continuous aggregates, which is faster than directly querying hypertables for the kind of aggregate queries reports require. Additionally, by not querying hypertables directly, you ensure higher insert performance (since you don’t have queries and inserts being performed on the same hypertable).For use cases like dashboarding and IT Ops reporting, where you want the speed of querying aggregated views and also need fully up-to-date data, check out our newreal-time aggregates feature.Q: Variables, once defined, are visible across the whole dashboard. Is there a way to make variables visible only to relevant panels?A:Grafana variables are dashboard-specific, not panel-specific, so multiple panels can access the same variable.This is useful if you have several panels that use the same variable, but can be tricky if your panel:variable ratio is closer to 1:1 (this can be the case with many dashboards that depend on a variety of data sources, like a CTO dashboard that looks at details for multiple products and deployments).There is no way to make variables apply only to specific panels, at the time of writing. However, there is anopen issue on GitHubfor you to add to and upvote.Two quick tips:A variablewon’tapply to a panel if it’s not used in the query, or as a parameter in the panel settings.If you find yourself with too many variables in one dashboard, consider branching off into smaller, more focused, and purpose-specific dashboards. This can help keep your dashboards organized and easy to navigate when you have many variables.Q: In the IT monitoring demo, your data was Prometheus metrics, stored in TimescaleDB. Can you explain a bit more about how TimescaleDB interacts with your Postgres table and Prometheus?A:Many people are familiar with using Prometheus as adirectGrafana datasource.But, in the demo, I use Prometheus to scrape metrics from my target (a database I’m monitoring), then use TimescaleDB as a remote read and write for Prometheus. This way, all of my Prometheus metrics are written to and stored in TimescaleDB, so I can query TimescaleDB for my metrics - rather than Prometheus directly.This is useful because:I have a long-term store for Prometheus data, so I can retain data over longer periods of timeI can aggregate metrics from multiple Prometheus instances in one database, giving me more operational ease (i.e., I simply query that one database v. multiple instances).Learn more about why you should use a long-term store for Prometheus metrics and how to get startedhere.Q: When are the next webinars in this series? Is there a way we can sign up to be notified?A:Glad you asked! The next webinar in our Grafana 101 series is “Getting Started with Alerting” (Wednesday, May 20th at 10 am PT/1 pm ET / 4 pm GMT). I hope to see you there....and to make sure you never miss sessions like this,signup for Timescale's newsletter, where we always add upcoming technical sessions (as well as announcements and best practices from our community).Ready for more Grafana goodness?As always, thank you to those who made it live – and to those who couldn’t, I – and the rest of Team Timescale – are here to help at any time. Reach out via our public Slack channel, and we’ll happily jump in.For easy reference, here are therecordingandslidesfor you to check out, re-watch, and share with friends and teammates.Want to learn more about this topic? Be sure to sign up for session 2 in our Grafana 101 technical session series, onWed, May 20: “Guide to Grafana 101: Getting Started with Alerts”🔍RSVPto learn how to use your Grafana dashboards to understand when issues arise in your critical metrics, get notifications, and take action quickly.To learn about future sessions and get updates about new content, releases, and other technical content, subscribe to ourBiweekly Newsletter.Hope to see you on the next one!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grafana-webinar-1-recap/
2022-09-09T12:44:52.000Z,Data Visualization Examples: How to Create Pie Charts in Grafana,"There are numerous data visualization examples in the glorious world of the Internet, but today we’re focusing on a good old classic sitting quietly in almost every data visualization dashboard: the pie chart.Pie charts are mostly used when dealing with group or categorized data. As the name suggests, the pie-shaped chart represents the whole data, divided into multiple parts or slices. Each piece of the pie represents a subcategory of the complete data.Pie charts can answer several questions:Which was the least traded stock volume last month?Which stock has the highest traded stock today?Who had the highest percentage of accrued votes in the last election?In this tutorial, you’ll learn what pie charts are and how to use them in the open-source analytics and interactive visualization web applicationGrafana—using PostgreSQL andTimescaleDB.Using a real-world stocks dataset, you’ll find a handful of data visualization examples on creating beautiful and insightful pie charts in Grafana (with just SQL!). We’ll cover creating pie and donut charts with pre-aggregated data using arbitrary time intervals (time_bucket) and show the transaction volume—all in the same panel.Join us along for the ride!Looking for more Grafana guides? Here are ourGrafana tutorialsand ourGrafana 101 Creating Awesome Visualizationsfor more support on visualizations in Grafana.If you need a database to store your time-series data and power your dashboards, try Timescale, our fast, easy-to-use, and reliable cloud-native data platform for time series built on PostgreSQL. (You can sign up for a 30-day free trial, no credit card required.)Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/data-visualization-examples-how-to-create-pie-charts-in-grafana/
2020-04-08T17:52:30.000Z,[New Webinar] Guide to Grafana 101: Getting started with (awesome) visualizations,"Get demos, queries, and expert tips to jumpstart your first dashboard, or add a little extra awesome to your existing Grafana setup.If you’re working with data, odds are you need a way to visualize your data - either for yourself or to share with your team, wider organization, or your end-users. There’s no shortage of tools available, and I’ve used quite a few: Tableau, Looker, PowerBI, and, my latest favorite, Grafana.So that we’re all on the same page, Grafana is an open-source visualization tool that’s growing in popularity with developers (including me!) for its wide variety of data sources, open source licensing, and support for many useful visualization types.But, using Grafana isn’t without a learning curve. In my conversations with developers, and in my own experiences, it can take weeks - if not months- to master the art of creating useful visualizations. There are a lot of features that you may not know how, when, or why to use, like filters, charts, gauges, and even world maps.What if you could shorten that time and get up and running with the visualizations you need straightaway (or add to ones that you’ve already built)?In my upcoming webinar,Guide to Grafana 101: Getting Started with (awesome) Visualizations,  I’ll show you how to use SQL and TimescaleDB to build Grafana charts, gauges, and beyond for any time-series dataset.I’ll focus on code and step-by-step live demos, using my own and my teammates’ time-series datasets to create various panels (what Grafana calls its charts and visualization types) and an example monitoring dashboard.Join me on April 22nd at 10am PT/1pm ET/4pm GMTwhere I’ll demo how to:Set up a datasource for GrafanaCreate 5+ different visualizations for data from IoT, infrastructure monitoring, and public datasetsCombine geo-spatial and time-series data in our dashboardsTake my demo and customize it for your project, team, or organizationExamples of the dashboards we’ll create from scratch during the sessionWhether you’ve never used Grafana and are looking for a cheaper alternative to proprietary visualization tools, or are a Grafana pro who’s looking for some other tips & tricks, this session is bound to teach you a thing (or two, or five).My goal is that you leave the session with the practical skills and inspiration you need to build your own (awesome) Grafana dashboards for your projects.RSVP here.As always, myself and other Timescale team experts will be available to answer questions throughout the session, and share ample resources and technical documentation.Signup even if you’re unable to attend live, and I’ll make sure you receive the recording, slides, and resources - and answer any questions you may have along the way.See you soon!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grafana-101-webinar/
2019-08-15T18:26:50.000Z,"Build an Application Monitoring Stack With TimescaleDB, Telegraf & Grafana","Match the flexibility and scale of your application with a stack that works for you.The world of systems design has become a powerful yet complex place. Through the advancement of microservice architectures, enterprise applications have become more fault tolerant, easier to scale, and capable of delivering end users with a constantly improving experience due to the ability for development teams to rapidly iterate and innovate.However, this presents the operations teams with some VERY complex challenges when it comes to monitoring the health and performance of these applications. For example:How do we implement a monitoring solution that matches the power and flexibility of the application we are deploying?How can we have an eye on all application layers and collect all the data we need to monitor key metrics that ensure the end user is having an optimal experience?The answer to those questions are simpler than you might think. We must look to anapplication monitoringstack that matches the flexibility and scale of the application we are monitoring.This monitoring stack can use best of breed components to instrument the needed data collection, and to store the data in such a way that is can be accessed quickly and can be used for both realtime and historical contexts. The end result provides us the ability to present the data based on the needs of the teams maintaining the application.In this blog, we’ll discuss such a stack using the following:Application Monitoring StackTimescaleDBTo collect the time series data that will be generated by as part of our monitoring, and provide a highly scalable and performant platform to store this data and make it available for both realtime and historical analysis.TelegrafTo instrument the data collection across the application.GrafanaTo provide the requisite needed visualization of the data points collected in TimescaleDB and allow us a window into the data being served by Timescale.Storing the data in TimescaleDBNow that you know what this application monitoring stack will look like, the first order of business is to make sure you have a TimescaleDB instance running (whether on premise or an instance running inManaged Service for TimescaleDB. TimescaleDB is the heart of the application monitoring stack, and is where the data from your application will land.If you are brand new to Timescale, follow the instructionshereto get TimescaleDB going locally or in the cloud.The nature of the data we are collecting is unique, as you can see from the sample below it represents time-series data:In the sample, we are capturing CPU metrics on a minute by minute basis, calling for a database technology that is purpose built to handle this type of data ingestion (large volume, high velocity), and obtains the requirements for querying this type of data. TimescaleDB is designed to help manage this type of data ingestion and complex queries.In our application monitoring use case we are going to look to cast the data in two ways:We need to be able to query the data in such a way that it will allow us to build real time dashboards, helping us understand what is happening “now”.We need to be able to store and query the data in a historical context, that is to say we need to be able to understand, and start to predict what will happen in the future (allowing us to prepare and/or budget for additional resources, and make the needed application level adjustments).Being able to serve these two use cases simultaneously is key since it represents the core of our application performance monitoring stack, and is at the core of the value provided by TimescaleDB.Another large benefit in using a technology like TimescaleDB is that it sits on top of PostgreSQL, making it simple to get the data out of the database. Whether you're using a tool like Grafana to build a dashboard (which we will discuss next) or want to run some ad hoc queries to understand trending in storage usage, the idea that you will be using standard SQL to do this reduces the learning curve and shortens the time to value of the entire stack.Instrumenting data collection with TelegrafThe next step is figuring out how we are going to collect data, and in this case we are going to use Telegraf and deploy it to all elements of our application.Telegraf gives us the facility to collect the information we need to properly monitor the health and performance of our application. We will be able to gather things like CPU, Memory, Network, as the basics (i.e. will operate at the pod level in a Kubernetes environment, and report pod based statistics if you have chosen this type of deployment). We can also instrument data collection / metrics that will help us evaluate things like database performance and application response times.The Telegraf agent is lightweight and simple to install, and will solve the first part of our problem: data collection.Using the linkhereyou will find step by step instructions for deploying Telegraf (Note: This version of Telegraf includes output plugin support for writing back to PostgreSQL and TimescaleDB).Configuring visualization & alerting with GrafanaFinally, let's talk about visualization and about how are we going to present the data we are collecting and use it to monitor what is happening across the application.In this case, we are going to use Grafana to help us understand the data in realtime while giving us the ability to set and trigger alarms when something is out of specification. (To set up Grafana visualization with TimescaleDB, follow theseinstructions.)As an example, we may choose to collect CPU data from the nodes in our application cluster, and we will want to make sure we monitor this in real time and set alarm thresholds to notify of a potential issue:We are capturing the real time data from the machine, monitoring both System and User CPU usage, and setting an alarm threshold when we reach 80% utilization. This is an example of data being evaluated and shown to the user in real-time. This particular dashboard is updated every 5 seconds.In contrast, we also need a higher level view of the same world. In the case below we are looking at the same set of CPU metrics but across a broader period of time (12 Hours):Again, being able to look at the larger picture as it relates to spotting trends is another use case we need to account for in this solution. The ability to pull back from the granular level, and view historical data is key to managing our application and its performance.In this case we are looking for spikes based on a particular window of our day (12 hours), however it is worth noting that because TimescaleDB is providing us with long term data storage we can pull back to a daily, weekly, or even monthly views of this data, with an eye on being proactive in spotting the trends associated with our application performance.Once this data is integrated into Grafana, we can define alert rules (e.g. “Average CPU usage greater than 80 percent for 5 minutes”). Once an alert is triggered, Grafana can dispatch a notification. (Instructions for setting up alerting in Grafana can be foundhere.)SummaryWhen it comes to implementing and application performance monitoring stack, flexibility along with best of breed components that are purpose built to carry out their part of the job is the key. Being able to understand the real time “situation” AND quickly and easily access historical trends is required.Our suggested stack will give you the flexibility to collect that data you are interested in, store it in a database that will serve the two key use cases (real-time and historical), provide you the flexibility around how you want to display the data, and ensure the data is collected and queried in a performant manner. All of this will make certain that you have access to the data you need to effectively manage your application.Have questions about setting up thisapplication monitoringstack? Reach out to us on our communitySlackchannel.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/build-an-application-monitoring-stack-with-timescaledb-telegraf-grafana/
2018-10-16T15:00:00.000Z,Grafana & TimescaleDB: Enhancing time-series exploration and visualization,"The new graphical query editor in Grafana v5.3 for PostgreSQL offers a visual interface and TimescaleDB compatibility.To further ourfirst-class integration with Grafana, we worked with their team to develop the new visual query editor, including TimescaleDB features (such astime_bucket), for thePostgreSQL datasource.Essentially, the query editor makes it easier for users to explore time-series data by improving the discoverability of data stored in PostgreSQL. Users can use drop-down menus to formulate their queries with valid selections and macros to express time-series specific functionalities, all without a deep knowledge of the database schema or the SQL language.Prior to the recently releasedGrafana v5.3, users had to handwrite SQL queries in order to query data. By combining the usability of the query editor with the power of full SQL, users can generate dynamic dashboards and visualizations in Grafana.Both the query editor and PostgreSQL datasource are contributions made byTimescaleDBengineers (including myself) to the open-source Grafana community. If you are interested in hearing more about why we decided to build a first-class integration with Grafana for PostgreSQL/TimescaleDB, and to learn more about the new capabilities now possible with the query editor, please read our post featured onGrafana’s blog.Additionally, my colleague (and Timescale Product Manager) Diana Hsieh, recently gave a live demo on how to use TimescaleDB and Grafana duringTime-Series Data NY. In the video, Diana provides an overview of Grafana and how it works with PostgreSQL and TimescaleDB, demoing the PostgreSQL/TimescaleDB connector, new graphical query builder, and TimescaleDB specific capabilities and functions that make storing time-series data in PostgreSQL not only possible, but quite powerful.Watch the video:You can learn more about our partnership with Grafana from our recentpress release. If you’re ready to get started, please download TimescaleDB (installation instructions) and Grafana (installation instructions).For more information about the new query editor, check out the recentGrafana v5.3 release notesand learn more from theirwebsite. To read more about TimescaleDB, check out ourwebsite,documentation, andGitHub.Want to stay updated with all thingsTimescale? Sign up for the community mailing list below.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grafana-time-series-exploration-visualization-postgresql-8c7baa9c3bfe/
2020-10-06T16:27:06.000Z,"Promscale: An analytical platform and long-term store for Prometheus, with the combined power of SQL and PromQL","⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Promscale is a horizontally scalable and operationally mature platform for Prometheus data that offers the combined power of PromQL and SQL, enabling developers to ask any question, create any dashboard, and achieve greater visibility into their systems. Promscale is built on top ofTimescaleDB, the leading relational database for time series.Promscaleis the result of a year of dedicated development effort by one of Timescale's engineering teams. It incorporates feedback from users and the general Prometheus community, and builds on 3.5 years of feedback from users of our previousPrometheus read-write adapter. As a result, despite being a young project, Promscale already sports an active user community, including organizations like Electronic Arts, Dow Chemical, and many others. This latest release marks the graduation of Promscale out of beta.(The name “Promscale” itself was picked by our users and the Prometheus community via thisGitHub poll. Although some of us were secretly rooting for “Promy McPromFace”😂.)To get started right away,visit our GitHub repoto install Promscale via Helm Charts, Docker, and others.And, if you like what we're building, please give us a ⭐️ on GitHub 🤗.If you have a Kubernetes cluster with Helm installed, we suggest usingtobsto install a full metric collection and visualization solution including Prometheus, Grafana, Promscale, and a preview version of PromLens in under 5 minutes (demovideo).Customers looking to migrate data from Prometheus into Promscale can useProm-migrator, an open-source, universal Prometheus data migration toolthat can move data from one remote-storage system to another.Note: Although Mat, Josh, and Harkishen are listed as the authors of this post, full credit goes to the entire Promscale team:Ante Krešić,Blagoj Atanasovski,David Kohn,Harkishen Singh,Josh Lockerman, andMat Arye.But why did we build Promscale? Please read on for more.We are witnessing a shift in the role of software, and in the ways organizations manage and monitor their softwareToday, every industry is moving its computing to the cloud. The complexity and scale of these modern, cloud-based applications necessitate sophisticated systems to monitor software application health and manage software infrastructure. Unlike in the past, when systems were all built using proprietary software, this new wave of modern infrastructure is being built using free, open components, like Kubernetes and Prometheus. The top two reasons for this shift are: flexibility and cost. Unlike proprietary SaaS solutions, open tools put the users' needs first, enabling them to customize their stack to meet their needs, and cost pennies on the dollar. In this world, developers, not sales contracts, nor RFPs, nor enterprise sales teams, decide which tools are used.Prometheus has emerged as the de facto monitoring solution for modern software systemsPrometheus, is an open-source systems monitoring and alerting toolkit that can be used to easily and cost-effectively monitor infrastructure and applications. Over the past few years, Prometheus has emerged as the monitoring solution for modern software systems. The key to Prometheus’ success is its pull-based architecture in combination with service discovery, which is able to seamlessly monitor modern, dynamic systems in which (micro-)services startup and shutdown frequently.Problem: Prometheus is not designed for analyticsAs organizations use Prometheus to collect data from more and more of their infrastructure, the benefits from mining this data also increase. Analytics becomes critical for auditing, reporting, capacity planning, prediction, root-cause analysis, and more. Prometheus's architectural philosophy is one of simplicity and extensibility. Accordingly, it does not itself provide durable, highly-available long-term storage or advanced analytics, but relies on other projects to implement this functionality.There are existing ways to durably store Prometheus data, but while these options are useful for long-term storage, they only support the Prometheus data model and query model (limited to the PromQL query language). While these work extremely well for the simple, fast analyses found in dashboarding, alerting, and monitoring, they fall short for more sophisticated analysis capabilities, or for the ability to enrich their dataset with other sources needed for insight-generating cross-cutting analysis.Solution: Promscale scales and augments Prometheus for long-term storage and analyticsEnter Promscale.We built Promscale to conquer a challenge that we, and other developers, know all too well: how do we easily find answers to complex questions in our monitoring data?Built on top ofTimescaleDBand PostgreSQL, Promscale supports both PromQL and SQL, offers horizontal scalability to over 10 million metrics per second and petabytes of storage, supportsnative compression,handles high-cardinality, provides rock-solid reliability, and more. It also offers other native time-series capabilities, such as data retention policies, continuous aggregate views, downsampling, data gap-filling, and interpolation. It is already natively supported by Grafana via thePrometheusandPostgreSQL/TimescaleDBdata sources.Promscale architecture and how it fits into the observability stackPrometheus writes data to the Promscale connector using theremote_writeAPI, storing the data in TimescaleDB. The Promscale connector understands PromQL queries natively and fetches data from TimescaleDB to execute them, while SQL queries go to TimescaleDB directly.Promscale is open-source, licensed under Apache 2. TimescaleDB is licensed under the completely free, source-availableTimescale License.Promscale stores data in a dynamically auto-generated schema highly optimized for Prometheus metrics that is the result of thorough benchmarking and community discussion. In particular, this schema decouples individual metrics, allowing for the collection of metrics with vastly different cardinalities and retention periods. At the same time, Promscale exposes simple, user-friendly views so that developers do not have to understand this optimized schema.Thanks to its relational foundation, Promscale also supports a variety of data types (numerics, text, arrays, JSON, booleans), JOINS, and ACID semantics, in addition to simple metric data. Because Promscale is built on top of PostgreSQL, it is operationally mature and includes capabilities such as high-availability, streaming backups, upgrades over time, roles and permissions, and security.Promscale also benefits from the TimescaleDB user community: tens of millions of downloads, over half a million active databases, 5,000+ member Slack channel.User testimonialsAlthough a relatively new project, Promscale is already in use by developers across the globe:""We have game metrics available in different data sources like Graphite, Datadog, and CloudWatch. We are storing all of these metrics in Prometheus, with Promscale for long-term storage. Promscale lets us collate metrics from these different sources and generate a single report in a unified view so that we can have better visibility into what is happening inside our games.""— Saket K., Software Engineer, Electronic Arts""Our goal is to have all of our sites from around the world monitored using Prometheus and view the resulting data in a user-friendly way. We chose Promscale to store our data because it scales, offers flexibility – for example, dividing read and write activities among different nodes – and has the operational maturity and rock-solid reliability of PostgreSQL, including streaming backups and high-availability.""—Adam B., Service Specialist, Dow ChemicalI was super skeptical at first (especially never having used Grafana w/ Postgres) but being able to still use#PromQLagainst@TimescaleDBis baller af— Matt (@halfmatthalfcat)August 13, 2020Install Promscale today via Helm Charts, Docker, and others.More information on GitHub.(And, if you like what we are building, please give us a ⭐️ on GitHub 🤗.)If you have a Kubernetes cluster with Helm installed, we suggest usingtobsto install a full metric collection and visualization solution including Prometheus, Grafana, Promscale, and a preview version of PromLens within 5 minutes (video).How to get involved with the Promscale community:For help with any technical questions, please joinTimescale Slack(#prometheus).To learn more about the origin, status, and roadmap for this project, please read on.Prometheus has emerged as the monitoring solution for modern software systemsOver the past few years,Prometheus, an open-source systems monitoring and alerting toolkit that can be used to easily and cost-effectively monitor infrastructure has emerged as the monitoring solution for modern software systems.Source: Prometheus docsThe key to Prometheus’ success is that it is built for modern, dynamic systems in which services start up and shut down frequently. The simple way that Prometheus collects data works extremely well with the ephemeral, churning nature of modern software architectures, and microservices in particular, because the services themselves don’t need to know anything about the monitoring system. Any service that wants to be monitored simply exposes its metrics over an HTTP endpoint. Prometheus scrapes these endpoints periodically and records the values it sees into a local time-series database.Prometheus’ decoupled architecture makes the system as a whole much more resilient. Services don’t need the monitoring stack to be up to get work done, and the monitoring software only needs to know about individual serviceswhile it’s actually scraping them. This makes it easy for the monitoring system to adjust seamlessly as services fail and new ones are brought up.This architecture also responds gracefully to overloading. While push-based architectures often drown in traffic when under high load. Prometheus simply slows down its scrape loop. Thus, while your metric resolution may suffer, your monitoring system will remain up and functional.Keeping with the theme of resilience and simplicity, Prometheus doesn’t try to store data for the long term, but rather exposes an interface allowing a dedicated database to do so instead. Prometheus continually pushes data to thisremote-writeinterface, ensuring that metric data is durably stored. That is where external long-term storage systems come in.Analytical options for Prometheus data are lackingAs developers use Prometheus to collect data from more and more of their infrastructure, the benefits from mining this data also increase. Analytics becomes critical for things like auditing, reporting, capacity planning, prediction, root-cause analysis, and more.Prometheus itself was developed with a clear sense of what it is, and is not, designed to do.Prometheus is designed to be a monitoring and alerting system; but it is not a durable, highly-available long-term store of data, nor a store for other datasets, nor a sophisticated analytics engine. However, though these capabilities are not provided by Prometheus itself, they are critical for the longer-duration and more intensive usages of metric data, including auditing, reporting, capacity planning, predictive analytics, root-cause analysis, and many others. As such, Prometheus provides hooks to forward its data to an external data store more suited for these tasks.Existing options for storing Prometheus data externally, while useful, all focus on long-term storage, and in some cases, limited forms of aggregation. Such systems can only store floats, and perform PromQL queries, making them too limited, both in data-stored and in query-model, to perform sophisticated analytics.In addition, as great as the Prometheus architecture is for recording data in highly dynamic environments, its method of collecting data at unaligned intervals creates challenges when analyzing data, since timestamps from multiple “simultaneous” scapes on different endpoints can differ by a significant amount.Prometheus devised a language called PromQL that addresses these difficulties by regularizing data at query time: aligning the data at user-specified intervals and discarding excess data points. While this method of analysis works extremely well for simple, fast analyses, found in dashboarding, alerting, and monitoring, it can be lacking for more-sophisticated analysis.For example, PromQL can’t aggregate across both series and time, making it quite difficult (if not impossible) to get accurate statistics over time for a particular label key, which is necessary for things such as determining when a memory leak was introduced by looking at 90th percentile memory usage grouped by app version across a long time-span. This kind of drill-down and reaggregation is important for many kinds of analytics, because even when the data contains the information needed for the problem at hand, it often wasn’t gathered with that kind of analysis in mind. Other PromQL features, such as joins, filters, and statistics, are similarly restricted, limiting its usage in discovering trends and developing insights.Others have also written about these issues: The CNCFSIG-Observability working grouphas put together alistof use cases in the observability space that need better tools for metrics analytics. Dan Luu, a popular tech blogger, also had a widely distributedblog postabout getting more value out of your metric data.This is where Promscale comes in.Why we built PromscaleWe say the market lacks a system for deep analytics of Prometheus data because we’ve felt that need while monitoring our own infrastructure. We built Promscale to conquer a challenge that we, and other developers, know all too well: how do we easily find answers to complex questions in our monitoring data?We are big fans of Prometheus as software developers and operators – in particular, we became involved in the Prometheus ecosystem 3.5 years ago when we initially published our previousPrometheus adapter, one of the first read-write adapters.But after multiple years of use and study we realized we needed capabilities beyond what Prometheus - and its associated tools - currently offer.In our stack, this includes things like:Support for distributed storage of datausingTimescaleDB multi-node deployment.Auxiliary data about the system being monitoredtoaugment metrics with additional information that helps us understand what they mean, such as node hardware properties, user/owner information, geographic location, or what the workload is running.Joinscombining metrics with this additional auxiliary data and metadata to create a complete view of the system.Efficient long-term storagefor historical analysis, such as reporting of past incidents, capacity planning, auditing, and more.Flexible data managementto handle the large volume of data monitoring generated, with tiering support such as multi-tenancy, automated data retention, and downsampling.Isolationbetween the various metrics.Since different metrics can be sent by completely different systems, we want both the performance and data management of different metrics to be independent (e.g., so that downsampling one metric won’t affect others).Logs and traces alongside metrics, to provide a better all-around view of the system. If all three modalities are in the same database, then JOINs between this data can lead to interesting insight. (To be clear, Promscale does not support logs and traces today, but this is an area of future work.)SQLas a versatile query language for those general analytics that PromQL isn’t suited for, as well as thelingua francaspoken by a variety of data analysis and machine learning tools.Fast and easy migration of existing datastorageusingProm-migrator, an open-source, universal Prometheus data migration tool.What our infrastructure team really wanted was an analytical platform on top of Prometheus to achieve more-insightful and cost-effective observability into our own infrastructure.That is what we built with Promscale.How Promscale worksArchitectureThis architecture uses the standard remote_write / remote_read Prometheus API, cleanly slotting into that space in the Prometheus stack.Prometheus writes data to the Promscale connector using theremote_writeAPI, storing the data in TimescaleDB. Promscale understands PromQL queries natively and fetches data from TimescaleDB to execute them, while SQL queries go to TimescaleDB directly.Promscale architecture and how it fits into the observability stackPromscale can be deployed in any environment running Prometheus, alongside any Prometheus instance. We provide Helm charts for easier deployments to Kubernetes environments.SQL interfaceThe data stored in Promscale can be queried both in PromQL and SQL. Though the data layout we use is internally quite sophisticated, you don’t need to understand any of it to analyze metrics through our easy-to-use SQL views.Each metric is exposed through a view named after the metric, so a measurement calledcpu_usageis queried like:SELECT 
	time, 
	value, 
	jsonb(labels) as labels 
FROM ""cpu_usage"";time                    value   labels  
2020-01-01 02:03:04	0.90   	{""namespace"": ""prod"", ""pod”: ""xyz""}
2020-01-01 02:03:05	0.98   	{""namespace"": ""dev"",  ""pod”: ""abc""}
2020-01-01 02:03:06	0.70   	{""namespace"": ""prod"", ""pod"": ""xyz""}The most important fields aretime,value, andlabels.labelsrepresents the full set of labels associated with the measurement and is represented as an array of identifiers. In the query above we view the labels in their JSON representation using thejsonb()function.Each row has aseries_iduniquely identifying the measurement’s label set. This enables efficient aggregation by series. You can easily retrieve the labels array from aseries_idusing thelabels(series_id)function. As in this query that shows how many data points we have in each series:SELECT
	jsonb(labels(series_id_)) as labels,
	count(*)
FROM ""cpu_usage"" 
GROUP BY series_id;labels               				count
{""namespace"": ""prod"", ""pod”: ""xyz""}		1
{""namespace"": ""dev"",  ""pod”: ""abc""}		7
{""namespace"": ""prod"", ""pod"": ""xyz""}		3Each label key (in our example namespace and pod) is expanded out into its own column storing foreign key identifiers to their value, which allows us to JOIN, aggregate and filter by label keys and values. You get back the text represented by a label id using theval(id)function. This opens up nifty possibilities such as aggregating across all series with a particular label key. For example, to determine the median CPU usage reported over the past year grouped by namespace, you could run:SELECT 
	val(namespace_id) as namespace, 
	percentile_cont(0.5) within group (order by value) 
AS median
FROM “cpu_usage” 
WHERE time > '2019-01-01'
GROUP BY namespace_id;namespace       median
prod            0.8
dev             0.7The complete view looks something like this:SELECT * FROM ""cpu_usage"";time			value	labels  series_id 	namespace_id	pod_id
2020-01-01 02:03:04	0.90    {1,2} 	1		1		2
2020-01-01 02:03:05	0.98    {4,5}	2		4		5
2020-01-01 03:03:06	0.70    {1,2)	1		1		1To simplify filtering by labels, we created operators corresponding to the selectors in PromQL. Those operators are used in aWHEREclause of the formlabels ? (<label_key> <operator> <pattern>). The four operators are:==matches tag values that are equal to the pattern!==matches tag value that are not equal to the pattern==~matches tag values that match the pattern regex!=~matches tag values that are not equal to the pattern regexThese four matchers correspond to each of the four selectors in PromQL, though they have slightly different spellings to avoid clashing with other PostgreSQL operators. They can be combined together using any boolean logic with any arbitrary WHERE clauses.For example, if you want only those metrics from the production namespace namespace or those whose pod starts with the letters ""ab"" you simply OR the corresponding label matchers together:SELECT avg(value) 
FROM ""cpu_usage"" 
WHERE labels ? ('namespace' == 'production') 
       OR labels ? ('pod' ==~ 'ab*')Combined, these features open up all kinds of possibilities for analytics. For example, you could get easily get the 99th percentile of memory usage per container in the default namespace with:SELECT 
  val(used.container_id) container, 
  percentile_cont(0.99) within group(order by used.value) percent_used_p99  
FROM container_memory_working_set_bytes used
WHERE labels ? ('namespace' == 'default')  
GROUP BY container 
ORDER BY percent_used_p99 ASC 
LIMIT 100;container             		       percent_used_p99
promscale-drop-chunk                            1433600
prometheus-server-configmap-reload              6631424
kube-state-metrics                             11501568Or, to take a more complex example fromDan Luu’s post, you can discover Kubernetes containers that are over-provisioned by finding those containers whose 99th percentile memory utilization is low:WITH memory_allowed as (
  SELECT 
    labels(series_id) as labels, 
    value, 
    min(time) start_time, 
    max(time) as end_time 
  FROM container_spec_memory_limit_bytes total
  WHERE value != 0 and value != 'NaN'
  GROUP BY series_id, value
)
SELECT 
  val(memory_used.container_id) container, 
  percentile_cont(0.99) 
    within group(order by memory_used.value/memory_allowed.value) 
    AS percent_used_p99, 
  max(memory_allowed.value) max_memory_allowed
FROM container_memory_working_set_bytes AS memory_used 
INNER JOIN memory_allowed
      ON (memory_used.time >= memory_allowed.start_time AND 
          memory_used.time <= memory_allowed.end_time AND
          eq(memory_used.labels,memory_allowed.labels)) 
WHERE memory_used.value != 'NaN'   
GROUP BY container 
ORDER BY percent_used_p99 ASC 
LIMIT 100;container			       percent_used_p99        total
cluster-overprovisioner-system    6.961822509765625e-05   4294967296
sealed-secrets-controller           0.00790748596191406   1073741824
dumpster                             0.0135690307617187    268435456Demo!In this 15-minute demo video, Avthar shows you how Promscale handles SQL and PromQL queries via the terminal and Grafana.Getting StartedInstall Promscale today via Helm Charts, Docker, and others.More information on GitHub.(And if you like what we are building, please give us a ⭐️ on Github 🤗.)If you are looking for managed Prometheus storage, get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.If you have a Kubernetes cluster with Helm installed, we suggest usingtobsto install a full metric collection and visualization solution including Prometheus, Grafana, Promscale, and a preview version of PromLens in under 5 minutes:Promscale can be deployed in any environment running Prometheus, alongside any Prometheus instance. If you already have Prometheus installed and/or aren’t using Kubernetes, see ourREADMEfor various installation options. Customers who have data in Prometheus already can migrate that data into Promscale usingProm-migrator, an open-source, universal Prometheus data migration toolthat can move data from one remote-storage system to another.How to get involved in the Promscale community:For help with any technical questions, please joinTimescale Slack(#prometheus).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/promscale-analytical-platform-long-term-store-for-prometheus-combined-sql-promql-postgresql/
2022-03-23T13:07:46.000Z,OpenTelemetry and Python: A Complete Instrumentation Guide,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.OpenTelemetry is considered by many the future of instrumentation, and it’s not hard to understand why. In a world where successful companies are software companies, the need to collect observations from running code is almost universal. Through metrics, logs, and traces, observability data gives us the information we need to inspect how our applications run—and thanks to projects like OpenTelemetry, observability is becoming accessible to everyone.In the past, collecting and analyzing observability data meant negotiating a challenging landscape: developers had to either buy into a walled garden provided by a commercial vendor or sacrifice interoperability by attempting to combine multiple open-source projects, each with a different instrumentation API and ecosystem.The open source path often resulted in combining components like Prometheus (metrics), Elastic (logs), and Jaeger (traces), but using many systems often felt complex with multiple instrumentation syntaxes, multiple outputs, multiple query languages, and multiple backends.OpenTelemetrypromises to solve this complexity by providing a vendor-agnostic standard for observability, allowing users to decouple instrumentation and routing from storage and query. The OpenTelemetry API (which defines how OpenTelemetry is used) and language SDKs (which define the specific implementation of the API for a language) generate observability data; this allows backends to be mixed and matched as needed, including Promscale, which aims to be a unified backend for all OpenTelemetry data.This approach allows OpenTelemetry users to concentrate on instrumentation as a separate concern. Users of OpenTelemetry can implement instrumentation without having to know where the data is going to be stored, what format it will be stored in, or how it will eventually be queried. As we will see below, developers can even take advantage ofauto-instrumentation: the codebase doesn’t need to be explicitly instrumented for a number of languages.OpenTelemetry Meets PythonAmong the three observability data types supported by OpenTelemetry (metrics, traces, and logs), traces are especially useful for understanding the behavior of distributed systems. OpenTelemetry tracing allows developers to create spans, representing a timed code block. Each span includes key-value pairs—called attributes—to help describe what the span represents, links to other spans, and events that denote timestamps within the span. By visualizing and querying the spans, developers gain a complete overview of their systems, helping them identify problems quickly when they arise.In this post, we will explore how we would instrument a Python application to emit tracing data (metric and log data interfaces are not stable quite yet). Then, we will examine:How auto-instrumentation of the same codebase works.The differences with manual instrumentation.How to mix manual instrumentation with auto-instrumentation.How to add information about exceptions.This guide focuses on Python code, but it is worth mentioning that OpenTelemetry offersinstrumentation SDKsfor many languages, like Java,JavaScript,Go,Rust, and more. In the case of auto-instrumentation, it is supported by a few languages (Python,Java,Node,Ruby, and.NET) with plans of adding more in the future.The Example Python AppWe will start with a supremely simple Python app that uses Flask to expose a route that models rolling a dice one or more times and summing the output. The default case rolls a 10- sided dice once, with request arguments that allow rolling extra times.from random import randint
from flask import Flask, request

app = Flask(__name__)

@app.route(""/roll"")
def roll():
    sides = int(request.args.get('sides'))
    rolls = int(request.args.get('rolls'))
    return roll_sum(sides,rolls)


def roll_sum(sides, rolls):
    sum = 0
    for r in range(0,rolls):
        result = randint(1,sides)
        sum += result
    return str(sum)Before we continue, let’s ensure that we configured our development environment correctly. You’ll need Python 3.x installed on your machine and Python pip to install packages. We will use a Python virtual environment to ensure our workspace is clean.To start, we will need to install Flask, which will also install the Flask binary, which we will use to run our app:mkdir otel-instrumentation
cd otel-instrumentation
python3 -m venv .
source ./bin/activate
pip install flaskNow that we have our environment ready, copy the code from above into a file calledapp.py.Adding OpenTelemetry Instrumentation ManuallyWe can run the Flask application using the flask command, then use curl to access the route, providing both a number of rolls and the number of sides for each dice. As expected, it will return the sum of the rolls (in this case, a single roll).In one terminal, run our Flask app:flask runAnd in another terminal, use curl to request the roll route from our Flask app:curl 'http://127.0.0.1:5000/roll?sides=10&rolls=1'To instrument this code, we need to add the following OpenTelemetry setup that will let us create traces and spans and export them to the console. Add this to the top of your Python app:from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.export import ConsoleSpanExporter

provider = TracerProvider()
processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)There are three concepts at play here, a provider, a processor, and a tracer.A provider (in this case TracingProvider) is the API entry point that holds configuration.A processor defines the method of sending the created elements (spans) onward.A tracer is an actual object which creates the spans.The code creates a provider, adds a processor to it, then configures the local tracing environment to use these.In our case, the processor is writing to the local console—you’ll probably note that this is hardcoded, which somewhat dilutes the OpenTelemetry mantra of removing downstream concerns from observability generation.If we wanted to send our traces elsewhere, we would need to alter this code. This is usually side-stepped by sending to an OpenTelemetry Collector—which can be thought of as a standalone proxy that receives inputs, processes them if needed, and exports them to one or more downstream locations. In our situation, we will stick with the console.We will also need to make sure that we have theopentelemetry-distro(which will pull in the SDK and the API, as well as make theopentelemetry-bootstrapandopentelemetry-instrumentcommands available) Python package installed by running the following command:pip install opentelemetry-distroOnce we have the components installed and the API configured, we can use the tracer object to add a span to our Flask route. Replace the app route with the following updated code:@app.route(""/roll"")
def roll():
    with tracer.start_as_current_span(""server_request""):
        sides = int(request.args.get('sides'))
        rolls = int(request.args.get('rolls'))
        return roll_sum(sides,rolls)That will make a span lasting for the length of the block on each call to the Flask route. If we stop-start the Flask application and access the endpoint again, we will see the OpenTelemetry tracing data written to the console.{
    ""name"": ""server_request"",
    ""context"": {
        ""trace_id"": ""0xae7539e266baff8ecab1c065f70839d3"",
        ""span_id"": ""0x760ae3a7cee38441"",
        ""trace_state"": ""[]""
    },
    ""kind"": ""SpanKind.INTERNAL"",
    ""parent_id"": null,
    ""start_time"": ""2022-03-11T02:27:08.090530Z"",
    ""end_time"": ""2022-03-11T02:27:08.090584Z"",
    ""status"": {
        ""status_code"": ""UNSET""
    },
    ""attributes"": {},
    ""events"": [],
    ""links"": [],
    ""resource"": {
        ""telemetry.sdk.language"": ""python"",
        ""telemetry.sdk.name"": ""opentelemetry"",
        ""telemetry.sdk.version"": ""1.9.1"",
        ""service.name"": ""unknown_service""
    }
}We have now created a span. We can see that there is not much helpful information other than the span ID (which will be referenced in child spans), the trace ID (which groups related spans), and the start and end times. We can add some attributes and an event per roll by changing the code again. Replace the route and the roll_sum definitions with the following:@app.route(""/roll"")
def roll():
    with tracer.start_as_current_span(
        ""server_request"", 
        attributes={ ""endpoint"": ""/roll"" } 
    ):

        sides = int(request.args.get('sides'))
        rolls = int(request.args.get('rolls'))
        return roll_sum(sides,rolls)

def roll_sum(sides, rolls):
    span = trace.get_current_span()
    sum = 0
    for r in range(0,rolls):
        result = randint(1,sides)
        span.add_event( ""log"", {
            ""roll.sides"": sides,
            ""roll.result"": result,
        })
        sum += result
    return  str(sum)We are adding an attribute to the span in the tracing start method, and in the roll_sum function, we obtain the current span and then add an event per roll. Events are commonly used like this to emit log events and hold exception information.We want to roll multiple times, so this time, we will pass rolls=2 via curl. You can use the following command:curl 'http://127.0.0.1:5000/roll?sides=10&rolls=2'The tracing output will now look something like this:{
    ""name"": ""server_request"",
    ""context"": {
        ""trace_id"": ""0x8c70746f079e4d03b4e23cf2b6886e5b"",
        ""span_id"": ""0x2505a6e946712b9c"",
        ""trace_state"": ""[]""
    },
    ""kind"": ""SpanKind.INTERNAL"",
    ""parent_id"": null,
    ""start_time"": ""2022-03-11T02:42:56.882101Z"",
    ""end_time"": ""2022-03-11T02:42:56.882243Z"",
    ""status"": {
        ""status_code"": ""UNSET""
    },
    ""attributes"": {
        ""endpoint"": ""/roll""
    },
    ""events"": [
        {
            ""name"": ""log"",
            ""timestamp"": ""2022-03-11T02:42:56.882216Z"",
            ""attributes"": {
                ""roll.sides"": 10,
                ""roll.result"": 3
            }
        },
        {
            ""name"": ""log"",
            ""timestamp"": ""2022-03-11T02:42:56.882231Z"",
            ""attributes"": {
                ""roll.sides"": 10,
                ""roll.result"": 4
            }
        }
    ],
    ""links"": [],
    ""resource"": {
        ""telemetry.sdk.language"": ""python"",
        ""telemetry.sdk.name"": ""opentelemetry"",
        ""telemetry.sdk.version"": ""1.9.1"",
        ""service.name"": ""unknown_service""
    }
}We can see an attribute has been set—letting us know the endpoint—and we can also see two events telling us the output of our two rolls and the time they were rolled.Instrumenting the App Automatically Using OpenTelemetry LibrariesLooking at the previous example, there are many attributes that could be added from the HTTP request. Rather than do this manually, we can use auto-instrumentation to do it standardly. Open Telemetry auto-instrumentation is instrumentation produced without code changes, often through monkey patching or bytecode injection. As previously mentioned, the feature only supports a few languages, so farPython,Java,Node,Ruby, and.NET(the latter requires some minimal code changes to enable auto-instrumentation).Auto-instrumentation isn’t omnipotent, nor is it as simple as a span per function. Instead, it’s custom implemented for a number of frameworks in a meaningful way. This is why you need to check if your language and framework are supported (Python Flask is, check out the otherson this GitHub repo!).We will need to install theopentelemetry-instrumentation-flaskpackage through our Python package manager to enable auto-instrumentation. Another option is runningopentelemetry-bootstrap -a install, which will install auto-instrumentation packages for all Python frameworks that support it.opentelemetry-bootstrap -a installTo run with auto-instrumentation, we pass the method we use to run the script (in this case, Flask run) and any arguments to theopentelemetry-instrumentcommand. We can do this with the original version of our code from the first example (that is, without any mention of OpenTelemetry).Copy the code into app.py and run Flask using this new method:opentelemetry-instrument --traces_exporter console \
 flask runWe are passing the processor using thetraces_exporterflag, which moves this configuration from the codebase into the application or container runtime. We will be outputting to the console again.Hit the endpoint with curl again, and you will see the following automatically generated tracing output:{
    ""name"": ""/roll"",
    ""context"": {
        ""trace_id"": ""0xa0e68f2c6febcc2ce392871264520cae"",
        ""span_id"": ""0x9731cb56a5d68bc2"",
        ""trace_state"": ""[]""
    },
    ""kind"": ""SpanKind.SERVER"",
    ""parent_id"": null,
    ""start_time"": ""2022-03-11T03:05:04.487693Z"",
    ""end_time"": ""2022-03-11T03:05:04.488611Z"",
    ""status"": {
        ""status_code"": ""UNSET""
    },
    ""attributes"": {
        ""http.method"": ""GET"",
        ""http.server_name"": ""127.0.0.1"",
        ""http.scheme"": ""http"",
        ""net.host.port"": 5000,
        ""http.host"": ""127.0.0.1:5000"",
        ""http.target"": ""/roll?sides=10&rolls=1"",
        ""net.peer.ip"": ""127.0.0.1"",
        ""http.user_agent"": ""curl/7.77.0"",
        ""net.peer.port"": 53603,
        ""http.flavor"": ""1.1"",
        ""http.route"": ""/roll"",
        ""http.status_code"": 200
    },
    ""events"": [],
    ""links"": [],
    ""resource"": {
        ""telemetry.sdk.language"": ""python"",
        ""telemetry.sdk.name"": ""opentelemetry"",
        ""telemetry.sdk.version"": ""1.9.1"",
        ""telemetry.auto.version"": ""0.28b1"",
        ""service.name"": ""unknown_service""
    }
}The output is similar to before, with a single trace containing a single span—but now we have a list of attributes that describe (from a Flask point of view) what is actually happening. We can see it was a GET request, via HTTP, to ‘/random,’ which came from the local 127.0.0.1 IP address. This comes for free—and has been defined in theopentelemetry-instrumentation-flaskpackage.This demonstrates the two key-value propositions of automatic instrumentation:OpenTelemetry supplies what they consider to be best practice implementations for frameworks, creating spans as needed. Such removes the need for a developer to try and decide on the relevant attributes to include in each span.OpenTelemetry configuration (like which processor to send to) can be injected without modifying your code. In this case, we are sending to the console, but this could be changed to Jaeger or an OpenTelemetry processor by modifying the command line arguments.Mixing Things Up: Combining Manual and Auto-InstrumentationBut what if we want to have a mix of auto-instrumentation and manual instrumentation?That’s possible, too. Let’s imagine we want to create a span for the roll_sum function and then attach the events from above. We can do this with a subset of the boilerplate we initially needed.How? Replace your app.rs with the following code:from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

from random import randint
from flask import Flask, request

provider = TracerProvider()
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

app = Flask(__name__)

@app.route(""/roll"")
def roll():
    sides = int(request.args.get('sides'))
    rolls = int(request.args.get('rolls'))
    return roll_sum(sides,rolls)


def roll_sum(sides, rolls):
    with tracer.start_as_current_span(""roll_sum""):  
        span = trace.get_current_span()
        sum = 0
        for r in range(0,rolls):
            result = randint(1,sides)
            span.add_event( ""log"", {
                ""roll.sides"": sides,
                ""roll.result"": result,
            })
            sum += result
        return  str(sum)We are now applying manual instrumentation to theroll_sumfunction as previously.That will create two spans: a parent representing the/rollroute (and is auto-implemented), one child representing theroll_sumfunction, and an event per roll attached. We have removed any reference to a processor from the setup code.And now, when we use the following command to rerun our application, it will be automatically injected.opentelemetry-instrument --traces_exporter console \
 flask runNow use curl to request the endpoint with two rolls:curl 'http://127.0.0.1:5000/roll?sides=10&rolls=2'And you will see output similar to the following emitted by the server to the console:{
    ""name"": ""roll_sum"",
    ""context"": {
        ""trace_id"": ""0x56822e51ee80474126f186a246f522d8"",
        ""span_id"": ""0xe6a230f1367bfc95"",
        ""trace_state"": ""[]""
    },
    ""kind"": ""SpanKind.INTERNAL"",
    ""parent_id"": ""0xdb33f044f5b450d5"",
    ""start_time"": ""2022-03-11T03:19:07.632525Z"",
    ""end_time"": ""2022-03-11T03:19:07.632589Z"",
    ""status"": {
        ""status_code"": ""UNSET""
    },
    ""attributes"": {},
    ""events"": [
        {
            ""name"": ""log"",
            ""timestamp"": ""2022-03-11T03:19:07.632563Z"",
            ""attributes"": {
                ""roll.sides"": 10,
                ""roll.result"": 4
            }
        },
        {
            ""name"": ""log"",
            ""timestamp"": ""2022-03-11T03:19:07.632578Z"",
            ""attributes"": {
                ""roll.sides"": 10,
                ""roll.result"": 3
            }
        }
    ],
    ""links"": [],
    ""resource"": {
        ""telemetry.sdk.language"": ""python"",
        ""telemetry.sdk.name"": ""opentelemetry"",
        ""telemetry.sdk.version"": ""1.9.1"",
        ""telemetry.auto.version"": ""0.28b1"",
        ""service.name"": ""unknown_service""
    }
}
{
    ""name"": ""/roll"",
    ""context"": {
        ""trace_id"": ""0x56822e51ee80474126f186a246f522d8"",
        ""span_id"": ""0xdb33f044f5b450d5"",
        ""trace_state"": ""[]""
    },
    ""kind"": ""SpanKind.SERVER"",
    ""parent_id"": null,
    ""start_time"": ""2022-03-11T03:19:07.630657Z"",
    ""end_time"": ""2022-03-11T03:19:07.632826Z"",
    ""status"": {
        ""status_code"": ""UNSET""
    },
    ""attributes"": {
        ""http.method"": ""GET"",
        ""http.server_name"": ""127.0.0.1"",
        ""http.scheme"": ""http"",
        ""net.host.port"": 5000,
        ""http.host"": ""127.0.0.1:5000"",
        ""http.target"": ""/roll?sides=10&rolls=2"",
        ""net.peer.ip"": ""127.0.0.1"",
        ""http.user_agent"": ""curl/7.77.0"",
        ""net.peer.port"": 53607,
        ""http.flavor"": ""1.1"",
        ""http.route"": ""/roll"",
        ""http.status_code"": 200
    },
    ""events"": [],
    ""links"": [],
    ""resource"": {
        ""telemetry.sdk.language"": ""python"",
        ""telemetry.sdk.name"": ""opentelemetry"",
        ""telemetry.sdk.version"": ""1.9.1"",
        ""telemetry.auto.version"": ""0.28b1"",
        ""service.name"": ""unknown_service""
    }
}We can see that ourroll_sumspan lists our route span as its parent!What About Exceptions?One other benefit that comes for free with auto-instrumentation is reporting information about exceptions. By changing the number of rolls in the curl command to a non-numeric value and hitting our auto-instrumented Flask server again, we will see a trace that contains an error event that describes the issue, including a traceback!Without stopping the server, use curl to request the endpoint with some bad data:curl 'http://127.0.0.1:5000/roll?sides=10&rolls=test'And you will see output similar to the following emitted by the server to the console:{
    ""name"": ""/roll"",
    ""context"": {
        ""trace_id"": ""0x465e01653dc7bda27572df93a6a17921"",
        ""span_id"": ""0x5626c188ddd93853"",
        ""trace_state"": ""[]""
    },
    ""kind"": ""SpanKind.SERVER"",
    ""parent_id"": null,
    ""start_time"": ""2022-03-17T05:47:48.163349Z"",
    ""end_time"": ""2022-03-17T05:47:48.171841Z"",
    ""status"": {
        ""status_code"": ""ERROR"",
        ""description"": ""ValueError: invalid literal for int() with base 10: 'test'""
    },
    ""attributes"": {
        ""http.method"": ""GET"",
        ""http.server_name"": ""127.0.0.1"",
        ""http.scheme"": ""http"",
        ""net.host.port"": 5000,
        ""http.host"": ""127.0.0.1:5000"",
        ""http.target"": ""/roll?sides=10&rolls=test"",
        ""net.peer.ip"": ""127.0.0.1"",
        ""http.user_agent"": ""curl/7.77.0"",
        ""net.peer.port"": 61368,
        ""http.flavor"": ""1.1"",
        ""http.route"": ""/roll"",
        ""http.status_code"": 500
    },
    ""events"": [
        {
            ""name"": ""exception"",
            ""timestamp"": ""2022-03-17T05:47:48.171825Z"",
            ""attributes"": {
                ""exception.type"": ""ValueError"",
                ""exception.message"": ""invalid literal for int() with base 10: 'test'"",
                ""exception.stacktrace"": ""Traceback (most recent call last):\n  File \""/Users/james/otel-instrumentation/lib/python3.8/site-packages/opentelemetry/trace/__init__.py\"", line 562, in use_span\n    yield span\n  File \""/Users/james/otel-instrumentation/lib/python3.8/site-packages/flask/app.py\"", line 2073, in wsgi_app\n    response = self.full_dispatch_request()\n  File \""/Users/james/otel-instrumentation/lib/python3.8/site-packages/flask/app.py\"", line 1518, in full_dispatch_request\n    rv = self.handle_user_exception(e)\n  File \""/Users/james/otel-instrumentation/lib/python3.8/site-packages/flask/app.py\"", line 1516, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \""/Users/james/otel-instrumentation/lib/python3.8/site-packages/flask/app.py\"", line 1502, in dispatch_request\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)\n  File \""/Users/james/otel-instrumentation/app.py\"", line 16, in roll\n    rolls = int(request.args.get('rolls'))\nValueError: invalid literal for int() with base 10: 'test'\n"",
                ""exception.escaped"": ""False""
            }
        }
    ],
    ""links"": [],
    ""resource"": {
        ""telemetry.sdk.language"": ""python"",
        ""telemetry.sdk.name"": ""opentelemetry"",
        ""telemetry.sdk.version"": ""1.10.0"",
        ""telemetry.auto.version"": ""0.29b0"",
        ""service.name"": ""unknown_service""
    }
}The VerdictOpenTelemetry is a fairly new technology that aims to consistently provide metrics, logs, and traces across implemented languages. It separates itself from the downstream storage and query layers, allowing these implementations to be mixed and matched or changed at a later date.As we have seen, the OpenTelemetry Python SDK provides both manual and automatic instrumentation options for traces, which can also be combined as needed. When you use auto-instrumentation with a supported framework, a predefined set of spans will be created for you and populated with relevant attributes (including error events when exceptions occur).If you want to get started with OpenTelemetry,check out Promscale, the observability backend built on PostgreSQL and TimescaleDB. Promscale has full support for OpenTelemetry traces, making it easy to store your traces in a relational database and analyze them with all the power and the convenience of SQL.If you want to learn more about Promscale:Visitour websiteand thePromscale docs.To get a taste of how much information you can get from your traces,take a look at our lightweight OpenTelemetry demo!Once you’re using Promscale, join the #promscale channel in our Community Slack. You will have the chance to interact directly with the team building the product and ask us any questions. You can also find us in our newCommunity Forum.Tune in toTimescale Community Dayon March 31 to hear from Ramon Guiu, VP of Observability at Timescale, about how to get insights about your distributed systems by ingesting OpenTelemetry traces into TimescaleDB using Promscale.See you there!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/opentelemetry-and-python-a-complete-instrumentation-guide/
2022-04-14T13:24:05.000Z,A Deep Dive Into the Four Types of Prometheus Metrics,"📚 Welcome to our series about metrics! In this first post, we deep-dived into the four types of Prometheus metrics; then,we examined how metrics work in OpenTelemetry;and finally, we put the two together—explaining the differences, similarities, and integration between the metrics in both systems.Metrics measure performance, consumption, productivity, and many other software properties over time. They allow engineers to monitor the evolution of a series of measurements (like CPU or memory usage, requests duration, latencies, and so on) via alerts and dashboards. Metrics have a long history in the world of IT monitoring and are widely used by engineers together with logs and traces to detect when systems don’t perform as expected.In its most basic form, a metric data point is made of:A metric nameThe timestamp when the data point was collectedA measurement represented by a numeric valueIn the last ten years, as systems have become increasingly complex, the concept of dimensional metrics, that is, metrics that also include a set of tags or labels (i.e., the dimensions) to provide additional context, emerged. Monitoring systems that support dimensional metrics allow engineers to easily aggregate and analyze a metric across multiple components and dimensions by querying for a specific metric name and filtering and grouping by label.For modern dynamic systems comprising many components,Prometheus, a Cloud Native Computing Foundation (CNCF) project, has become the most popular open-source monitoring software and is effectively the industry standard for metrics monitoring.Prometheus defines ametric exposition formatand a remote write protocol that the community and many vendors have adopted to expose and collect metrics, becoming a de facto standard.OpenMetricsis another CNCF project that builds upon the Prometheus exposition format to offer a vendor-agnostic, standardized model for the collection of metrics that aims to be part of the Internet Engineering Task Force (IEFT).More recently, another CNCF project,OpenTelemetry, has emerged with the goal of providing a new standard that unifies the collection of metrics, traces, and logs, enabling easier instrumentation and correlation across telemetry signals.With a few different options to pick from, you may be wondering which standard is best for you. To help you answer this question, we have prepared a three-part blog post series in which we will be diving deep into the metric standards hosted by the CNCF.In this first post, we will cover Prometheus metrics; in the next one, we willreview OpenTelemetry metrics; and in the final blog post,we will directly compare both formats—providing some recommendations for better interoperability.Our hope is that after reading these blog posts, you will understand the differences between each standard, so you can decide which one would best address your current (and future) needs.Prometheus MetricsFirst things first. There are four types of metrics collected by Prometheus as part of its exposition format:CountersGaugesHistogramsSummariesPrometheus uses apull modelto collect these metrics; that is, Prometheus scrapes HTTP endpoints that expose metrics. Those endpoints can benatively exposed by the componentbeing monitored or exposed via one of thehundreds of Prometheus exportersbuilt by the community. Prometheus providesclient libraries in different programming languagesthat you can use to instrument your code.The pull model works great when monitoring a Kubernetes cluster, thanks toservice discoveryand shared network access within the cluster, but it’s harder to use to monitor a dynamic fleet of virtual machines, AWS Fargate containers or Lambda functions with Prometheus. Why?It’s difficult to identify the metrics endpoints to be scraped, and access to those endpoints may be limited by network security policies. To solve some of those problems, the community released thePrometheus Agent Modeat the end of 2021, which only collects metrics and sends them to a monitoring backend using the remote write protocol.Prometheus can scrape metrics in both the Prometheus exposition and the OpenMetrics formats. In both cases, metrics are exposed via HTTP using a simpletext-basedformat (more commonly used and widely supported) or a more efficient and robust protocol buffer format. One big advantage of the text format is that it is human-readable, which means you can open it in your browser or use a tool like curl to retrieve the current set of exposed metrics.Prometheus uses a very simple metric model with four metric types that are only supported in the client libraries. All the metric types are represented in the exposition format using one or a combination of a single underlying data type. This data type includes a metric name, a set of labels, and a float value. The timestamp is added by the monitoring backend (Prometheus, for example) or an agent when they scrape the metrics.Each unique combination of a metric name and set of labels defines a series while each timestamp and float value defines a sample (i.e., a data point) within a series.Some conventions are used to represent the different metric types.A very useful feature of the Prometheus exposition format is the ability to associate metadata to metrics to define their type and provide a description. For example, Prometheus makes that information available, and Grafana uses it to display additional context to the user that helps them select the right metric and apply the right PromQL functions:Metrics browser in Grafana displaying a list of Prometheus metrics and showing additional context about them.Example of a metric exposed using the Prometheus exposition format:# HELP http_requests_total Total number of http api requests
# TYPE http_requests_total counter
http_requests_total{api=""add_product""} 4633433# HELPis used to provide a description for the metric and# TYPEa type for the metric.Now, let's get into more detail about each of the Prometheus metrics in the exposition format.CountersCounter metrics are used for measurements that only increase. Therefore they are always cumulative—their value can only go up. The only exception is when the counter is restarted, in which case its value is reset to zero.The actual value of a counter is not typically very useful on its own. A counter value is often used to compute the delta between two timestamps or the rate of change over time.For example, a typical use case for counters is measuring API calls, which is a measurement that will always increase:# HELP http_requests_total Total number of http api requests
# TYPE http_requests_total counter
http_requests_total{api=""add_product""} 4633433The metric name ishttp_requests_total, it has one label namedapiwith a value ofadd_productand the counter’s value is4633433. This means that theadd_productAPI has been called 4,633,433 times since the last service start or counter reset. By convention, counter metrics are usually suffixed with_total.The absolute number does not give us much information, but when used with PromQL’sratefunction (or a similar function in another monitoring backend), it helps us understand the requests per second that API is receiving. The PromQL query below calculates the average requests per second over the last five minutes:rate(http_requests_total{api=""add_product""}[5m])To calculate the absolute change over a time period, we would use a delta function which in PromQL is called increase():increase(http_requests_total{api=""add_product""}[5m])This would return the total number of requests made in the last five minutes, and it would be the same as multiplying the per second rate by the number of seconds in the interval (five minutes in our case):rate(http_requests_total{api=""add_product""}[5m]) * 5 * 60Other examples where you would want to use a counter metric would be to measure the number of orders in an e-commerce site, the number of bytes sent and received over a network interface or the number of errors in an application. If it is a metric that will always go up, use a counter.Below is an example of how to create and increase a counter metric using the Prometheus client library for Python:from prometheus_client import Counter
api_requests_counter = Counter(
                        'http_requests_total',
                        'Total number of http api requests',
                        ['api']
                       )
api_requests_counter.labels(api='add_product').inc()Note that since counters can be reset to zero, you want to make sure that the backend you use to store and query your metrics will support that scenario and still provide accurate results in case of a counter restart.GaugesGauge metrics are used for measurements that can arbitrarily increase or decrease. This is the metric type you are likely more familiar with since the actual value with no additional processing is meaningful and they are often used. For example, metrics to measure temperature, CPU, and memory usage, or the size of a queue are gauges.For example, to measure the memory usage in a host, we could use a gauge metric like:# HELP node_memory_used_bytes Total memory used in the node in bytes
# TYPE node_memory_used_bytes gauge
node_memory_used_bytes{hostname=""host1.domain.com""} 943348382The metric above indicates that the memory used in nodehost1.domain.comat the time of the measurement is around 900 megabytes. The value of the metric is meaningful without any additional calculation because it tells us how much memory is being consumed on that node.Unlike when using counters,rateanddeltafunctions don’t make sense with gauges. However, functions that compute the average, maximum, minimum, or percentiles for a specific series are often used with gauges. In Prometheus, the names of those functions areavg_over_time,max_over_time,min_over_time, andquantile_over_time. To compute the average of memory used onhost1.domain.comin the last ten minutes, you could do this:avg_over_time(node_memory_used_bytes{hostname=""host1.domain.com""}[10m])To create a gauge metric using the Prometheus client library for Python you would do something like this:from prometheus_client import Gauge
memory_used = Gauge(
                'node_memory_used_bytes',
                'Total memory used in the node in bytes',
                ['hostname']
              )
memory_used.labels(hostname='host1.domain.com').set(943348382)HistogramsHistogram metrics are useful to represent a distribution of measurements. They are often used to measure request duration or response size.Histograms divide the entire range of measurements into a set of intervals—named buckets—and count how many measurements fall into each bucket.A histogram metric includes a few items:A counter with the total number of measurements. The metric name uses the_countsuffix.A counter with the sum of the values of all measurements. The metric name uses the_sumsuffix.The histogram buckets are exposed as counters using the metric name with a_bucketsuffix and ale labelindicating the bucket upper inclusive bound. Buckets in Prometheus are inclusive, that is a bucket with an upper bound of N (i.e.,le label) includes all data points with a value less than or equal to N.For example, the summary metric to measure the response time of the instance of theadd_productAPI endpoint running onhost1.domain.comcould be represented as:# HELP http_request_duration_seconds Api requests response time in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_sum{api=""add_product"" instance=""host1.domain.com""} 8953.332
http_request_duration_seconds_count{api=""add_product"" instance=""host1.domain.com""} 27892
http_request_duration_seconds_bucket{api=""add_product"" instance=""host1.domain.com"" le=""0""}
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""0.01""} 0
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""0.025""} 8
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""0.05""} 1672
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""0.1""} 8954
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""0.25""} 14251
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""0.5""} 24101
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""1""} 26351
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""2.5""} 27534
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""5""} 27814
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""10""} 27881
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""25""} 27890
http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com"", le=""+Inf""} 27892The example above includes thesum, thecount, and 12 buckets. Thesumandcountcan be used to compute the average of a measurement over time. In PromQL, the average duration for the last five minutes will be computed as follows:rate(http_request_duration_seconds_sum{api=""add_product"", instance=""host1.domain.com""}[5m]) / rate(http_request_duration_seconds_count{api=""add_product"", instance=""host1.domain.com""}[5m])It can also be used to compute averages across series. The following PromQL query would compute the average request duration in the last five minutes across all APIs and instances:sum(rate(http_request_duration_seconds_sum[5m])) / sum(rate(http_request_duration_seconds_count[5m]))With histograms, you can compute percentiles at query time for individual series as well as across series. In PromQL, we would use thehistogram_quantilefunction. Prometheus uses quantiles instead of percentiles. They are essentially the same thing but quantiles are represented on a scale of 0 to 1 while percentiles are represented on a scale of 0 to 100. To compute the 99th percentile (0.99 quantile) of response time for theadd_productAPI running onhost1.domain.com, you would use the following query:histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{api=""add_product"", instance=""host1.domain.com""}[5m]))One big advantage of histograms is that they can be aggregated. The following query returns the 99th percentile of response time across all APIs and instances:histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))In cloud-native environments, where there are typically many instances of the same component running, the ability to aggregate data across instances is key.Histograms have three main drawbacks:First, buckets must be predefined, requiring some upfront design. If your buckets are not well defined, you may not be able to compute the percentiles you need or would consume unnecessary resources. For example, if you have an API that always takes more than one second, having buckets with an upper bound (le label) smaller than one second would be useless and just consume compute and storage resources on your monitoring backend. On the other hand, if 99.9 % of your API requests take less than 50 milliseconds, having an initial bucket with an upper bound of 100 milliseconds will not allow you to accurately measure the performance of the API.Second, they provide approximate percentiles, not accurate percentiles. This is usually fine as long as your buckets are designed to provide results with reasonable accuracy.And third, since percentiles need to be calculated server-side, they can be very expensive to compute when there is a lot of data to be processed. One way to mitigate this in Prometheus is to userecording rulesto precompute the required percentiles.The following example shows how you can create a histogram metric with custom buckets using the Prometheus client library for Python:from prometheus_client import Histogram
api_request_duration = Histogram(
                        name='http_request_duration_seconds',
                        documentation='Api requests response time in seconds',
                        labelnames=['api', 'instance'],
                        buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 25 )
                       )
api_request_duration.labels(
    api='add_product',
    instance='host1.domain.com'
).observe(0.3672)SummariesLike histograms, summary metrics are useful to measure request duration and response sizes.A summary metric includes these items:A counter with the total number of measurements. The metric name uses the_countsuffix.A counter with the sum of the values of all measurements. The metric name uses the_sumsuffix. Optionally, a number of quantiles of measurements exposed as a gauge using the metric name with a quantile label. Since you don’t want those quantiles to be measured from the entire time an application has been running, Prometheus client libraries use streamed quantiles that are computed over a sliding time window (which is usually configurable).For example, the summary metric to measure the response time of the instance of theadd_productAPI endpoint running onhost1.domain.comcould be represented as:# HELP http_request_duration_seconds Api requests response time in seconds
# TYPE http_request_duration_seconds summary
http_request_duration_seconds_sum{api=""add_product"" instance=""host1.domain.com""} 8953.332
http_request_duration_seconds_count{api=""add_product"" instance=""host1.domain.com""} 27892
http_request_duration_seconds{api=""add_product"" instance=""host1.domain.com"" quantile=""0""}
http_request_duration_seconds{api=""add_product"" instance=""host1.domain.com"" quantile=""0.5""} 0.232227334
http_request_duration_seconds{api=""add_product"" instance=""host1.domain.com"" quantile=""0.90""} 0.821139321
http_request_duration_seconds{api=""add_product"" instance=""host1.domain.com"" quantile=""0.95""} 1.528948804
http_request_duration_seconds{api=""add_product"" instance=""host1.domain.com"" quantile=""0.99""} 2.829188272
http_request_duration_seconds{api=""add_product"" instance=""host1.domain.com"" quantile=""1""} 34.283829292This example above includes the sum and count as well as five quantiles. Quantile 0 is equivalent to the minimum value and quantile 1 is equivalent to the maximum value. Quantile 0.5 is the median and quantiles 0.90, 0.95, and 0.99 correspond to the 90th, 95th, and 99th percentile of the response time for theadd_product APIendpoint running onhost1.domain.com.Like histograms, summaries include sum and count that can be used to compute the average of a measurement over time and across time series.Summaries provide more accurate quantiles than histograms but those quantiles have three main drawbacks:First, computing the quantiles is expensive on the client-side. This is because the client library must keep a sorted list of data points overtime to make this calculation. The implementation in the Prometheus client libraries uses techniques that limit the number of data points that must be kept and sorted, which reduces accuracy in exchange for an increase in efficiency. Note that not all Prometheus client libraries support quantiles in summary metrics. For example, the Python library does not have support for it.Second, the quantiles you want to query must be predefined by the client. Only the quantiles for which there is a metric already provided can be returned by queries. There is no way to calculate other quantiles at query time. Adding a new quantile requires modifying the code and the metric will be available from that time forward.And third and most important, it’s impossible to aggregate summaries across multiple series, making them useless for most use cases in dynamic modern systems where you are interested in the view across all instances of a given component. Therefore, imagine that in our example theadd_productAPI endpoint was running on ten hosts sitting behind a load balancer. There is no aggregation function that we could use to compute the 99th percentile of the response time of theadd_productAPI endpoint across all requests regardless of which host they hit. We could only see the 99th percentile for each individual host. Same thing if instead of the 99th percentile of the response time for theadd_productAPI endpoint we wanted to get the 99th percentile of the response time across all API requests regardless of which endpoint they hit.The code below creates a summary metric using the Prometheus client library for Python:from prometheus_client import Summary
api_request_duration = Summary(
                        'http_request_duration_seconds',
                        'Api requests response time in seconds',
                        ['api', 'instance']
                       )
api_request_duration.labels(api='add_product', instance='host1.domain.com').observe(0.3672)The code above does not define any quantile and would only produce sum and count metrics. The Prometheus client library for Python does not have support for quantiles in summary metrics.Histograms or Summaries, What Should I Use?In most cases, histograms are preferred since they are more flexible and allow for aggregated percentiles.Summaries are useful in cases where percentiles are not needed and averages are enough, or when very accurate percentiles are required. For example, in the case of contractual obligations for the performance of a critical system.The table below summarizes the pros and cons of histograms and summaries.Table comparing different properties of histograms vs. summaries in Prometheus.ConclusionIn the first part of this blog post series on metrics, we’ve reviewed the four types of Prometheus metrics: counters, gauges, histograms, and summaries. In the next part of the series,we will dissect OpenTelemetry metrics.If you're looking for a time-series database to store your metrics, check out Timescale. You will specially love it if you're using PostgreSQL. Timescale a PostgreSQL extension that will give PostgreSQL the boost it needs to handle large volumes of metrics,will keeping your writes and queries fastviaautomatic partitioning, query planner enhancements, improved materialized views,columnar compression, and much more.If you're running your PostgreSQL database in your own hardware,you can simply add the TimescaleDB extension. If you prefer to try Timescale in AWS,create a free account on our platform. It only takes a couple seconds, no credit card required!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/four-types-prometheus-metrics-to-collect/
2021-10-11T13:02:56.000Z,"What Are Traces, and How SQL (Yes, SQL) and OpenTelemetry Can Help Us Get More Value Out of Traces to Build Better Software","⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Developing software is hard. Debugging complex software systems is often harder. When our software systems are not healthy, we need ways to quickly identify what is happening, and then dig deeper to understand why it is happening (and then fix the underlying issues). But in modern architectures based on microservices, Kubernetes and cloud, identifying problems (let alone predicting them) has become more and more difficult.Enter observability, built on the collection of telemetry from modern software systems. This telemetry typically comes in the form of three signals: metrics, logs, and traces.Metrics and logs are well known and have been widely adopted for many years through tools likeNagios,Prometheus, or theELKstack.Traces, on the other hand, are relatively new and have seen much lower adoption. Why? Because, for most engineers, tracing is still a relatively new concept. Because getting started takes a lot of manual instrumentation work. And because, once we have done that work,getting value out of trace datais hard (for example, most tracing tools today just provide the ability to look up a trace by id or apply very simple filtering like Jaeger or Grafana Tempo).However, traces are key to understanding the behavior of and troubleshooting modern architectures.Here we demystify traces and explain what they are, and why they are useful, using a concrete example. Then we describe OpenTelemetry and explain how it vastly simplifies the manual instrumentation work required to generate traces.Finally, we announce the beta release of trace support in Promscale, the observability backend powered by SQL, via OpenTelemetry, and describe how Promscale and SQL enable us to get much more value out of trace data.Read on for more. If you’d like to get started with Promscale right away:Install the latest version of Promscale, following the instructions in ourGitHub repository(appreciate any GitHub stars!). As a reminder, Promscale is open-source and completely free to use.Join the TimescaleDB Slack community, where you’ll find 7K+ developers and Timescale engineers active in all channels. (The dedicated #promscale channel has 2.9K+ members, so it’s a great place to connect with like-minded community members, ask questions, get advice, and more).If you would like to connect with us, catch our talks atPromCon North America. We are also atKubeCon+CloudNativeCon North America🎉 Come to say hi to the Timescale booth (#S76)or join the Timescale virtual booth to chat with Promscale engineers, see Promscale in action during Office Hours session (Oct 14, 10:30 am PDT), and get cool swag!What Is a Trace?A “trace” represents how an individual request flows through the various microservices in a distributed system. A “span” is the fundamental building block of a trace, representing a single operation. In other words, a trace is a collection of underlying spans.Let’s look at a specific example to illustrate the concept.Imagine that we have a news site that is made of four micro-services:Thefrontendservice, which serves the website to our customer’s browser.Thenewsservice, which provides the list of articles from our news database that is populated by the editorial system (and has the ability to search articles and get individual articles together with their comments).Thecommentservice, which lets you save new comments in the comment database and retrieve all comments for an article.Theadvertisingservice, which returns a list of ads to display from a third-party provider.An architecture diagram for an example news site made up of four microservices: a frontend service, a news service, a comment service, and an advertising service.When a user clicks on the link to view an article on her browser this is what happens:1) Thefrontendservice receives the request.2) Thefrontendservice calls thenewsservice passing the identifier for the article.3) Thenewsservice calls the news database to retrieve the article.4) Thenewsservice calls thecommentservice to retrieve the comments for the  article.5) Thecommentservice calls the comment database and runs a query to retrieve all comments for the article.6) Thenewsservice gets all comments for the article and sends the article and the comments back to thefrontendservice.7) Thefrontendservice calls theadvertisingservice passing the contents of the article.8) Theadvertisingservice makes a REST API request to the third-party ads provider with the contents of the article to retrieve the list of optimized ads to display.9) Thefrontendservice builds the HTML page including the news, the comments and the ads send the response back to the user’s browser.When adding trace instrumentation to each of those services, you generate spans for each operation in the execution of the request outlined above (and more if you want more detailed tracking). This would result in a hierarchy of spans like shown in the diagram below:Hierarchy of spans for the news site microservice architecture. The length of each span indicates their duration.The entry point for the request is thefrontendservice. The first span thefrontendservice emits covers the entire execution of the request. That span is called the root span. All other spans are descendants of that root span. The length of each span indicates their duration.What do traces show?As you can see from the diagram above there are two key pieces of information we can extract from a trace:A connected representation of all the steps to process individual requests which very easily allow us to zero-in on the service and operation that is causing issues in your system when troubleshooting a production problem.The dependencies between different components in the system which we could use to build a map of how those components connect to each other. In a large system with tens or hundreds of components it is important to explore and understand the topology of the system to identify problems and improvements.Faster troubleshooting with tracesLet’s see through a practical example of how traces help identify problems in your applications faster.Continuing with our example, imagine that less than 1% of the REST API requests to the ads provider are suffering from slow response time (30 seconds). But those requests always come from the same set of users who are complaining to your support team and on Twitter.We look into the problem but aggregate percentile metrics (see our blog post onpercentile metricsto learn how to use them) are not indicating any problem because overall performance (99th percentile of response time, p99 for short) is great since this only impacts a small number of requests.Then we start looking into our logs but that’s a daunting task. We have to look at logs for each individual service and there are concurrent requests that make it very hard to read and connect those logs to identify the problem.What we need is to reconstruct requests around the time users reported the issue, find the requests that were slow, and then determine where that slowness happened.With logs, we have to do that manually which for a high traffic site it would take hours to do.With traces, we can do that automatically and quickly identify where the problem is.One simple way to do it is to search for the slowest traces, that is, top ten root spans with the highest duration during the time the users complained and look at what’s consuming most of the execution time. In our example, by looking at the visual trace representation we would very quickly see that what’s common in the slowest traces is that the request to the ads provider REST API is taking too long. An even better way (if your tracing system allows for that) would be to run a query to retrieve the slowest spans in the execution path of the slowest traces which would return the spans tracking the REST API calls.Note that looking at the p99 response time of those REST API calls would not have revealed any problem either because the problem occurred in less than 1% of those requests.We’ve quickly narrowed down where the problem is and can start looking for a solution as well as inform the ads provider of the issue so they can investigate it.One quick fix could be to put in place a one-second timeout in the API request so that a problem with the ads provider doesn’t impact your users. Another more sophisticated solution could be to make ads rendering an asynchronous call so it doesn’t impact rendering the article.Traces help us proactively make software betterWe can also use traces to proactively make our software better. For example, we could search for the slowest spans that represent database requests to identify queries to optimize. We could search for requests (i.e., traces) that involve a high number of services, or many database calls, or too many external service calls, and look for ways to optimize them and/or simplify the architecture of the system.RecapWe have seen how traces are useful to troubleshoot problems in microservices environments faster and discover improvements that we could implement to help us build better software that delights our customers.So what’s needed to get the benefits that traces provide? A tool to instrument services to generate traces, and a system to store, query and visualize them.Continue reading to learn how OpenTelemetry makes instrumentation easier, and how Promscale and SQL enable us to get more value out of traces, faster.Traces in OpenTelemetryOpenTelemetryis a vendor-agnostic emerging standard to instrument and collect traces (and also metrics and logs!) that can then be sent to and analyzed in any OpenTelemetry compatible backend. It has recently beenaccepted as a CNCF incubating projectand it has a lot of momentum: it’sthe second most active CNCF project, only after Kubernetes, with contributions fromall major observability vendors, cloud providers, and many end users(including Timescale).OpenTelemetry includes a number of core components:API specification: defines how to produce telemetry data in the form of traces, metrics, and logs.Semantic conventions: defines a set of recommendations to standardize the information to include in the telemetry (for example attributes like the status code of a span representing an http request) and ensure better compatibility between systems.OpenTelemetry protocol (OTLP): defines a standard encoding, transport, and delivery mechanism of telemetry data between the different components of an observability stack: telemetry sources, collectors, backends, etc.SDKs: language-specific implementations of the OpenTelemetry API with additional capabilities like processing and exporting for traces, metrics and logs.Instrumentation libraries: language-specific libraries that provide instrumentation for other libraries. All instrumentation libraries support manual instrumentation and several offer automatic instrumentation through byte-code injection.Collector: this component provides the ability to receive telemetry from a wide variety of sources and formats, process it and export it to a number of different backends. It eliminates the need to manage multiple agents and collectors.An architecture diagram illustrating the core components of the OpenTelemetry collector, the inputs it accepts and possible outputs it can produce.To collect traces from our code with OpenTelemetry we will use the SDKs and instrumentation libraries for the language your services are written in. Instrumentation libraries make OpenTelemetry easy to adopt because they can auto-instrument (yes, auto-instrument!) our code for services written in languages that allow for injecting instrumentation at runtime like Java and Node.js.For example, the OpenTelemetry Java instrumentation library automatically collects traces from alarge number of libraries and frameworks. For languages where that is not possible (like Go) we also get libraries that simplify the instrumentation but require more changes to the code.Even more, developers of libraries and components are already adding OpenTelemetry instrumentation directly in their code. Two examples of that areKubernetesandGraphQL Apollo Server.Once our code is instrumented we have to configure the SDK to export the data to an observability backend. While we can send the data directly from our application to a backend, it is more common to send the data to the OpenTelemetry Collector and then have it send the data to one or multiple backends. This way you can simplify the management and configuration of where you want to send the data and have the possibility to do additional processing (downsampling, dropping, transforming data) before it is sent to another system for analysis.Anatomy of an OpenTelemetry traceTheOpenTelemetry tracing specificationdefines the data model for a trace. Technically, a trace is adirected acyclic graphof Spans with parent-child relationships. Traces must include an operation name, start and finish timestamps, a parent Span identifier and the SpanContext. The SpanContext contains the TraceId, the SpanID, TraceFlags and the Tracestate. Optionally, Spans can also have a list of Events, Links to other related Spans and a set of Attributes (key-value pairs). Events are typically used to add to a Span one-time events like errors (error message, stacktrace and error code) or log lines that were recorded during the span execution. Links are less common but allow OpenTelemetry to support special scenarios like relating spans that are included in a batch operation with the batch operation span which would be initiated by multiple parents (i.e., all the individual spans that add elements to the batch).Attributes can contain any key-value pair.Trace semantic conventionsdefine some mandatory and optional attributes to help with interoperability. For example, a span representing the client or server of anHTTP requestmust have an http.method attribute and may have http.url and http.host attributes.Data model of an OpenTelemetry SpanRecapOpenTelemetry is a new vendor-agnostic standard that makes trace instrumentation and collection much easier through automation, SDKs and a protocol to provide interoperability with observability tools.Analyzing Traces With SQLAbove, we talked about several ways to get value out of traces. For example, when debugging our news application, searching for the top 10 root spans with the highest duration during the time that users complained. Or, when trying to proactively improve our news application, to search for the slowest spans that represent database requests to identify queries to optimize. Or similarly, to search for requests (i.e. traces) that involve a high number of services, or many database calls, or too many external service calls, and look for ways to optimize them and/or simplify the architecture of the system.Yet there is no standard way to analyze trace data to ask these questions. In the past, a small number of organizations (e.g., Google, Twitter, Uber, etc.) have built their own systems to do deeper analysis on their trace data. There have also been a number of open-source efforts (e.g., Jaeger, Zipkin, Grafana Tempo) which provide a UI that is very helpful to find and visualize individual traces but don’t provide the flexibility of a language to run any query and aggregate traces as needed to derive new insights.Turns out that we already have a universal query language for data analysis, one that most developers (and many non-developers) already know: SQL.In fact, with SQL, we can interrogate our trace data to answer any question we need to answer, in a way not possible with existing open-source tracing tools like Jaeger and Grafana Tempo.For example:List the operations with the highest error rates.List the slowest API methods.List the slowest database queries across all services.Identify customers suffering from the worst database query performance and how it compares to performance across all customers.Identify the upstream service causing the load on a service that is seeing elevated load.Even more, with SQL you could correlate traces and metrics at query time. For example, you could show response time per service correlated with CPU and memory consumption per service if you have container metrics in the same database.But of course, before we can use SQL to query traces (and metrics), we need to be able to store those traces in a scalable system that supports SQL, yet is designed for observability.Enter Promscale.Promscale Is the Observability Backend Powered by SQLPromscale was first announcedone year agoas an analytical platform for Prometheus metrics. Promscale is built on the solid foundation of TimescaleDB and PostgreSQL, and therefore has full SQL support(along with many other neat featureslike 100% PromQL compliance). Today, companies like Digital Ocean, Electronic Arts and Catalytic rely on Promscale to scale Prometheus to do long-term retention and analysis on their metrics.Our vision for Promscale is to enable engineers to store all observability data (metrics, logs, traces, metadata, and other future data types) in a single mature and scalable store and analyze it through a unified and complete SQL interface that provides developers with:Broad support for observability standards (e.g., OpenTelemetry, Prometheus/PromQL, StatsD, Jaeger, Zipkin, etc.) to simplify integration in any environment.Operational simplicity with just one storage system that is easy to deploy, manage, scale, and learn about.A familiar experience with PostgreSQL as the foundation and unified access to all data via full SQL support so they don’t need to learn other query languages.Unparalleled insights through the power of TimescaleDB’s advanced time-series analytical functions and Postgres’ SQL query capabilities (joins, sub-queries, etc.) to analyze and correlate observabilityandbusiness data.100s of out-of-the-box integrations through the PostgreSQL ecosystem: visualization tools, IDEs, ORMs, management tools, performance tuning, etc.Today we are announcing the beta release of trace support in Promscale, our second major step in fulfilling our vision.Promscale exposes an ingest endpoint that is OTLP-compliant which makes integration with OpenTelemetry instrumentation straightforward. Other tracing formats like Jaeger, Zipkin or OpenCensus can also be sent to Promscale through the OpenTelemetry Collector. Traces stored in Promscale can be queried with full SQL and visualized using Jaeger or Grafana.Architecture diagram illustrating Promscale architecture, with inputs from Prometheus metrics and OpenTelemetry traces, TimescaleDB as the core database powering Promscale and outputs to a variety of tools including Jaeger and Grafana.Designing an optimized schema for tracesPromscale stores OpenTelemetry traces using the following schema:Promscale schema for tracesThe schema is heavily influenced by the OpenTelemetryprotocol buffer definitions.The heart of the model is the span table. The span table is ahypertable, using the power of TimescaleDB to manage the ingestion and querying of spans over time. Similarly, both links and events are hypertables. Link tables use the start time of the source span, whereas each event has its own associated time.The span table is self-referencing. All the spans in a given trace form a tree. The span table uses the adjacency model to capture these trees - each span contains the id of the span’s parent. The root span of each trace has aparent_span_idof 0. The OpenTelemetry protocol buffers already utilize an adjacency model to represent the tree structure, so keeping this same model at the database layer makes ingestion easier. Other models such as the path enumeration model can be derived from an adjacency model, and we do this in several convenience functions.Spans, links, events, and resources can all be decorated with zero or moreattributes. Attributes are key-value pairs in which the key is a text name, and the value can be a primitive type (string, boolean, double, or 64 bit integer), or a homogeneous array.We have chosen to use “tag” instead of “attribute” in our model. “Attribute” gets to be a real pain to type over and over and over, and by our estimation “tag” is more widely used and understood in the industry.Tag values map very closely to json, and therefore we have chosen to store tag values in thejsonb data type. PostgreSQL has extensive support for storing andmanipulatingjsonb values, includingindexingandjsonpath querying. By using jsonb, we can piggy-back on PostgreSQL’s features to provide rich ways of filtering spans that in many cases will be indexed operations.Many tags will be repeated again and again across many spans. For this reason, we are not storing tags directly in the span table, but normalizing them out intotag_keyand tag tables. This eliminates data duplication and greatly reduces the storage required for the span table.Traditionally, a many-to-many relationship between spans and tags would be represented with an additional mapping table. We decided to eschew this approach. An additional mapping table would have considerably more records than the span table itself. It would need to contain both thetrace_idandspan_idwhich are 128 bits and 64 bits respectively, a timestamptz (it would need to be a hypertable in its own right), and the 64 bit ids of the tag andtag_key. This would require significant additional storage.Instead of this mapping table, we created a domain over jsonb called a “tag_map”. Atag_mapis a json object where keys are the ids of thetag_keytable, and the associated values are ids of the tag table. Thus, thetag_mapis a set of key-value pairs oftag_keysto tag values. We have thus essentially collapsed the mapping table into the span table.The tag table may in some setups become quite large. We have utilized PostgreSQL’sdeclarative partitioningto hash partition the tag table on the key text over 64 partitions. This effectively “load balances” the tag values over many smaller tables rather than storing them all in one big table. By hash partitioning on key text, all the values for a given key will be colocated in the same partition, and thereby belong to the same partition-level indexes, improving performance. Partitioning the tags should also provide some resiliency, in that operations on one key should not impact all keys. The number 64 was chosen more or less randomly. We intend to do some testing to find an optimal number of partitions.We are sharing how our schema works to help other developers in the community looking at solving a similar problem. Knowing the design of the underlying schema can also be helpful in situations where you may need to improve the performance of some query. However, you don’t really need (and we don’t expect you!) to understand how our schema works because Promscale comes with out-of-the-box views, functions and operators that make querying the data easier and faster that we cover in the next section.Querying Traces using SQLPromscale provides unified access to all your traces and metrics through a single, robust and well-known query language: SQL. Thanks to the power of SQL, the out-of-the-box views, functions and operators built-in in Promscale and the time-series analytical functions provided by TimescaleDB you can interrogate your data about pretty much any question you need to answer.Everything you need to query traces is in theps_tracedatabase schema:3 views: span, event and link. Those views automatically join the different tables in the schema to provide a consolidated view of a span, an event or a link as if all attributes were stored in the same table.A number of operators so you can easily apply conditions to resource, span, link and event attributes (we call those attributes tags in our data model).Functions to easily navigate through and retrieve tag values.To understand how to query traces we will use some of the examples we listed at the beginning of this section. The data we use in the examples comes from a Kubernetes cluster runningHoneycomb’s forkofGoogle’s microservices demothat uses OpenTelemetry instrumentation.In the examples below we show the raw results from the SQL queries. Those results can be easily displayed in Grafana dashboards by connecting to the underlying TimescaleDB/PostgreSQL database using Grafana’s PostgreSQL datasource:Grafana dashboard showing performance metrics from querying traces in Promscale with SQLQuery 1:  List the top operations with the highest error rate in the last hourResponse time, throughput and error rate are the key metrics used to assess the health of a service. In particular, high error rate and response time are key indicators of the experience we are delivering to our users and need to be tracked closely. In this example, we look at how we can use OpenTelemetry traces and SQL to identify the main sources of errors in our services.Every span has an attribute that indicates the name of the service (service_name) and an attribute that indicates the name of the operation (name). Technically, the service name in OpenTelemetry is not a span attribute but a resource attribute. For the purposes of querying traces in Promscale we can think of service name as a span attribute as well.Every span has another attribute that indicates whether the span resulted in an error or not:status_code. If there is an error, the value ofstatus_codeiserror.Our goal is to write a query that will return one row per individual operation with the total number of executions (number of spans), total number of executions that led to an error (spans where the status code is an error) and the percentage of spans with an error for the top 10 operations with more errors:SELECT
    service_name,
    span_name as operation,
    COUNT(*) FILTER (WHERE status_code = 'STATUS_CODE_ERROR') as spans_with_error,
    COUNT(*) as total_spans,
    TO_CHAR(100*(CAST(COUNT(*) FILTER (WHERE status_code = 'STATUS_CODE_ERROR') AS float) / count(*)), '999D999%') as error_rate
FROM span
WHERE
     start_time > NOW() - INTERVAL '1 hour'
GROUP BY service_name, operation
ORDER BY error_rate DESC
LIMIT 10;As you can see, the query uses the standard SQL syntax we are all familiar with:SELECT,FROM,WHERE,GROUP BY,ORDER BY.In theSELECTclause we project the service name and the operation (which corresponds to the span name). These are also the two attributes we use to aggregate the results in theGROUP BYclause.The other three columns we project are the number of spans with an error, the total number of spans and the error rate within each error group. To do it we use some nifty SQL capabilities available in PostgreSQL:COUNT (*)which returns the total number of rows, spans in this case, in a group.COUNT (*) FILTERwhich returns the total number of spans in a group matching a certain criteria. In our case we want spans with an error which are indicated by the valueerrorin thestatus_codeattribute.CASTto convert the number of spans with error to a floating point number so when it’s divided by the total count of spans it returns a floating number and not an integer. If we don’t do this then the number will be converted to the closest integer which will be 0 since that division is always less than 0.TO_CHARto convert the error rate to an easy to read and understand percentage number.Below is an example of results from this query:service_name |             operation             | spans_with_error | total_spans | error_rate
--------------+-----------------------------------+------------------+-------------+------------
 frontend     | /cart/checkout                    |               12 |         345 |    3.478%
 frontend     | hipstershop.AdService/GetAds      |               60 |        5214 |    1.115%
 adservice    | hipstershop.AdService/GetAds      |                1 |        5214 |     .019%
 cart         | grpc.health.v1.Health/Check       |                0 |         707 |     .000%
 cart         | hipstershop.CartService/EmptyCart |                0 |         345 |     .000%
 cart         | hipstershop.CartService/GetCart   |                0 |        7533 |     .000%
 checkout     | SQL SELECT                        |                0 |         361 |     .000%
 checkout     | grpc.health.v1.Health/Check       |                0 |         718 |     .000%
 checkout     | getDiscounts                      |                0 |         345 |     .000%
 checkout     | hipstershop.CartService/EmptyCart |                0 |         345 |     .000%Which immediately indicates that we need to take a closer look at the code behind the/cart/checkoutoperation since it has a very high error rate most likely leading to many lost sales!Another thing that drives our attention in these results are the second and third rows. What’s surprising about them is that the error rate on the client side of the request(frontend - hipstershop.AdService/GetAds) is much higher than the error rate on the server side of the same request (adservice - hipstershop.AdService/GetAds). So the adservice is successfully completing the request but something is happening when transferring the response back to the frontend service.Before we move on to the next example, let’s look at a different way to write the query that leverages additional SQL capabilities. To calculate the error rate in the query above we are using twice the functions required to count spans with error and total spans. This can lead to inconsistencies if we update the query to change the way we calculate the error rate but don’t apply those changes everywhere. To avoid that we can use SQL subqueries:SELECT
  service_name,
  operation,
  spans_with_error,
  total_spans,
  TO_CHAR(100*(CAST(spans_with_error AS float)) / total_spans, '999D999%') as error_rate
FROM (
    SELECT
        service_name,
        span_name as operation,
        COUNT(*) FILTER (WHERE status_code = 'error') as spans_with_error,
        COUNT(*) as total_spans
    FROM span
    WHERE
        start_time > NOW() - INTERVAL '1 hour'
    GROUP BY service_name, operation
) AS error_summary
ORDER BY error_rate DESC
LIMIT 10;This query first builds a dataset (error_summary) with service name, operation, spans with error and total spans and then it uses the values calculated in that dataset to compute the error rate avoiding the duplication in the initial query. This change doesn’t impact the performance of the query.This is a straightforward example of what you can do with subqueries. SQL subqueries provide a lot of flexibility for analyzing your traces to derive new insights.An additional consideration is that the query searches across all spans and depending on the environment and the amount of instrumentation you could see some duplicative results because of parent-child relationships between spans. In those cases it could be better to start by looking at traces (complete request) that had an error. You can do that by only searching across root spans by adding an additional condition to the where clause:WHERE start_time > NOW() - INTERVAL '1 hour' AND parent_span_id = 0Query 2: List the top slowest operations in the last hourAt the beginning of the previous example we identified response time and error rate as two key indicators of the experience we deliver to our users. Let’s see now how we can quickly determine bottlenecks in our services.All spans contain a duration attribute that indicates how long it took to execute that span. To analyze the duration we will look at several statistics and in particular percentiles. To learn more about percentiles and why you should use them instead of averages take a look atour blog post on this subject.In this example, our goal is to write a query that will return one row per individual operation with several percentiles (99.9th, 99th, 95th) and the average of the duration in milliseconds. For this one we will search across root spans so we see which user requests have the worse response time:SELECT
    service_name,
    span_name as operation,
    ROUND(approx_percentile(0.999, percentile_agg(duration_ms))::numeric, 3) as duration_p999,
    ROUND(approx_percentile(0.99, percentile_agg(duration_ms))::numeric, 3) as duration_p99,
    ROUND(approx_percentile(0.95, percentile_agg(duration_ms))::numeric, 3) as duration_p95,
    ROUND(avg(duration_ms)::numeric, 3) as duration_avg
FROM span
WHERE
    start_time > NOW() - INTERVAL '1 hour' AND
    parent_span_id = 0
GROUP BY service_name, operation
ORDER BY duration_p99 DESC
LIMIT 10;This query uses TimescaleDB’sapprox_percentilehyperfunctionto calculate the different percentiles. You could write the same query using PostgreSQL’s nativeprecentile_contfunction. However,approx_percentileis faster thanpercentile_contwhile incurring a small error (less than 3%). With our test data, approx_percentile performs 35% faster thanpercentile_contwith minimal error (less than 3%). If you need more precision, you can decrease the error at the expense of lower performance by replacingpercentile_aggwithuddsketch.We use theROUNDfunction to limit the number of decimals shown in each column. We need to use::numericto convert the return value ofapprox_percentilewhich is double precision to a type supported by theROUNDfunction, numeric in this case.service_name |              operation              | duration_p99 | duration_p999 | duration_p95 | duration_avg
--------------+-------------------------------------+--------------+---------------+--------------+--------------
 frontend     | /cart/checkout                      |    20658.359 |     20658.359 |     1238.359 |      600.609
 adservice    | AdService.start                     |    13319.743 |     13319.743 |    13319.743 |    13307.008
 cart         | grpc.health.v1.Health/Check         |     1843.628 |      1843.628 |      911.864 |      132.689
 frontend     | /                                   |     1159.651 |      1159.651 |      673.087 |      286.100
 frontend     | /product/{id}                       |      752.567 |       752.567 |      307.197 |      118.877
 frontend     | /cart                               |      582.594 |       582.594 |      270.289 |       92.472
 checkout     | hipstershop.CurrencyService/Convert |      193.229 |       193.229 |      101.888 |       30.177
 frontend     | /setCurrency                        |        7.504 |         7.504 |        0.908 |        0.498
 payment      | grpc.grpc.health.v1.Health/Check    |        1.615 |         1.615 |        0.172 |        0.386
 currency     | grpc.grpc.health.v1.Health/Check    |        1.355 |         1.355 |        0.140 |        0.420If we look at the results we can for example quickly see that there seems to be an issue with the /cart/checkout endpoint. Note that if we just looked at the average we would think performance is good (600 ms). Even if we look at the 95% percentile performance still looks acceptable at 1.2 seconds. However, when we look at the 99% percentile we can see the performance is extremely poor (20 seconds). So somewhere in between 1% and 5% of the requests delivered a very poor user experience. This is definitely something worth investigating further.As we can see, the results of the query are showing the performance of automated health checks. That may be useful in some scenarios but it’s not something our users experience and we may want to filter them out. To do it we would just add an additional condition to the where clause:start_time > NOW() - INTERVAL '1 hour' AND
parent_span_id = 0 AND
span_name NOT LIKE '%health%'Query 3: Identify what services generate more load on other services by operation in the last 30 minutesIn microservice architectures, there are many internal calls between services. One of those microservices could be going through a lot of load. That may be a service that is used by several other services for performing different operations and we would not immediately know what’s causing that.Using the query below we can quickly list all dependencies across services and get an understanding of not only who is calling who and how often, but also what are the specific operations involved in those calls and how long the execution of those calls is taking in aggregate (in seconds). This serves as an additional indicator of load since some types of requests could be much more expensive than others.SELECT
    client_span.service_name AS client_service,
    server_span.service_name AS server_service,
    server_span.span_name AS server_operation,
    count(*) AS number_of_requests,
    ROUND(sum(server_span.duration_ms)::numeric) AS total_exec_time
FROM
    span AS server_span
    JOIN span AS client_span
    ON server_span.parent_span_id = client_span.span_id
WHERE
    client_span.start_time > NOW() - INTERVAL '30 minutes' AND
    client_span.service_name != server_span.service_name
GROUP BY
    client_span.service_name,
    server_span.service_name,
    server_span.span_name
ORDER BY
    server_service,
    server_operation,
    number_of_requests DESC;In this query we are leveraging another powerful capability of SQL: joins. We are joining the span table with itself to identify only the spans that represent a call between two services. This is what the conditionclient_span.service_name != server_span.service_nameaccomplishes.The result would look something like the following:client_service | server_service |                    server_operation                     | number_of_requests | total_exec_time
----------------+----------------+---------------------------------------------------------+--------------------+-----------------
 frontend       | adservice      | hipstershop.AdService/GetAds                            |               2672 |               1
 frontend       | cart           | hipstershop.CartService/AddItem                         |                509 |               5
 checkout       | cart           | hipstershop.CartService/EmptyCart                       |                174 |               2
 frontend       | cart           | hipstershop.CartService/GetCart                         |               3697 |              25
 checkout       | cart           | hipstershop.CartService/GetCart                         |                174 |               2
 frontend       | checkout       | hipstershop.CheckoutService/PlaceOrder                  |                174 |              57
 frontend       | currency       | grpc.hipstershop.CurrencyService/Convert                |               8635 |              14
 checkout       | currency       | grpc.hipstershop.CurrencyService/Convert                |                408 |               1
 frontend       | currency       | grpc.hipstershop.CurrencyService/GetSupportedCurrencies |               3876 |               3
 checkout       | email          | /hipstershop.EmailService/SendOrderConfirmation         |                174 |               0
 checkout       | payment        | grpc.hipstershop.PaymentService/Charge                  |                174 |               0
 frontend       | productcatalog | hipstershop.ProductCatalogService/GetProduct            |              20436 |               1
 checkout       | productcatalog | hipstershop.ProductCatalogService/GetProduct            |                234 |               0
 frontend       | productcatalog | hipstershop.ProductCatalogService/ListProducts          |                501 |               0
 frontend       | recommendation | /hipstershop.RecommendationService/ListRecommendations  |               3374 |              15
 frontend       | shipping       | hipstershop.ShippingService/GetQuote                    |               1027 |               0
 checkout       | shipping       | hipstershop.ShippingService/GetQuote                    |                174 |               0
 checkout       | shipping       | hipstershop.ShippingService/ShipOrder                   |                174 |               0These results show the client service making the request in the first column and the service and operation receiving the request in the second and third columns. We see for example that the GetProduct method of the ProductCatalogService has been called from the frontend service and the checkout service and that the former made many more calls in the last 30 minutes, which is expected and not an issue in this case. But if we saw a much higher percentage of calls to the GetProduct method originating from the checkout service this would be an indicator of something unexpected going on.ConclusionTraces are extremely useful to troubleshoot and understand modern distributed systems. They help us answer questions that are impossible or very hard to answer with just metrics and logs. Adoption of traces has been traditionally slow because trace instrumentation has required a lot of manual effort and existing observability tools have not allowed us to query the data in flexible ways to get all the value traces can provide.This is not true anymore thanks to OpenTelemetry and Promscale.OpenTelemetry is quickly becomingtheinstrumentation standard and it offers libraries that automate (or at the very least simplify) trace instrumentation dramatically reducing the amount of effort required to instrument our services. Additionally, the instrumentation is vendor agnostic and the traces it generates can be sent to any compatible observability backend so we can change backends or use multiple ones.Promscale is the observability backend with full SQL support for querying traces and metrics. With Promscale you can query your data to answer any question you need to answer.The combined power of OpenTelemetry and Promscale help you get more value out of traces and build better software.Get started with PromscaleTo start getting more value out of your traces and metrics with Promscale:Check out our tracing documentationfor more details on how to start collecting traces with OpenTelemetry, Jaeger, and Zipkin and how to visualize them in Jaeger and Grafana.Learn about Promscale installationfor more on how Promscale works with Prometheus, installation instructions, sample PromQL and SQL queries, and more.Check ourGitHub repository. As a reminder, Promscale is open-source and completely free to use. (GitHub ⭐️  welcome and appreciated! 🙏)Promscale is also available on Timescale.Get started now with free 30-day trial (no credit card required).WatchPromscale 101 YouTube playlistfor step-by-step demos and best practices.Whether you’re new to Promscale or an existing community member, we’d love to hear from you!Join TimescaleDB Slack, where you’ll find 7K+ developers and Timescale engineers active in all channels. (The dedicated#promscalechannel has 2.5K+ members, so it’s a great place to connect with like-minded community members, ask questions, get advice, and more).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/what-are-traces-and-how-sql-yes-sql-and-opentelemetry-can-help-us-get-more-value-out-of-traces-to-build-better-software/
2021-05-19T13:33:04.000Z,Promscale 0.4: Drawing Inspiration from Cortex to Rewrite Support for Prometheus High-Availability,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Now with better support for Prometheus high-availability, support for multi-tenancy, improved user permissions (using role-based access control), and more.Last October,we released an early version of Promscale, an open-source analytical platform and long-term store for Prometheus data. Promscale grew out of a need to have something to store metrics and handle both simple dashboarding queries as well as answer more complex questions. For example, complex questions that involve other datasets and analytical queries, for use cases like capacity planning, predictive analytics, root-cause analysis, auditing, and reporting.Because of our architectural decision to build on top of TimescaleDB (and, thus, on PostgreSQL), Promscale is the only Prometheus long-term store that offers full SQL and fullPromQLsupport, providing the ability to create any dashboard and ask any question to get a more complete understanding of the systems they are monitoring. (This includes the ability to include external datasets and query across them using joins.)Despite being a relatively young project, Promscale users include two of the largest cloud providers in North America and Europe, one of the largest gaming companies in the world, many smaller startups, and a range of many other use cases and industries (including Timescale, where this project is also being developed - “dogfooding” FTW).Today, withPromscale 0.4, we’re introducing several new capabilities:Better support for Prometheus high-availability (more below)Support for multi-tenancyMore control over user permissions (using role-based access control)And more (please see release notes)With this release, Promscale is now even more robust and suited for simple and enterprise-grade deployments (and everything in between). In particular, our improved support for Prometheus high-availability will enable scalable deployments that are resilient to failures and outages while multi-tenancy provides better isolation of data belonging to different parts of your organization.In the rest of this post, we’ll go deeper into how we’ve redesigned (and subsequently improved) the way Promscale ingests data from Prometheus servers running as high-availability (HA) replicas.To get started with Promscale today:Install Promscale via Helm Charts, Docker, and others.See GitHub for more information(GitHub ⭐️ welcome and appreciated! 🙏)Check out ourGetting Started with Promscale tutorialfor more on how Promscale works, installation instructions, sample PromQL and SQL queries, and more.Watch ourPromscale 101 YouTube playlistfor step-by-step demos and best practice.Reach out inTimescale Slack(#promscale) to join our community.If you’re an existing user, simply deploy the latest version of the Promscale binary to upgrade to Promscale 0.4.IntroductionIn general, deploying Prometheus high-availability replicas is critical for robust production systems since they protect against a crash of any one server. Promscale has supported ingesting this kind of data since our first release – but our original method was based on database locks, which led to complex deployments, had problems with scalability and coupling, and was less resilient to certain kinds of failures.Our new system, which takes inspiration fromCortex, solves these issues and makes Promscale both easier to use and more robust.To the average reader, this may seem strange: why discuss how a feature implementation of another Prometheus project inspired our own?We believe that this type of knowledge sharing illustrates one of the most powerful parts of the Prometheus ecosystem: due to its open-source nature, and the strong collaborative culture within the Prometheus community, projects often take inspiration from one another, together improving the ecosystem as a whole.In this case, we designed our approach using some of the same basic principles as Cortex; conversely, aspects ofourimplementation may be useful for Cortex or other long-term storage systems for Prometheus metrics to use as inspiration to improve their own projects. This virtuous cycle improves all projects in the ecosystem.Read on to learn more about: how Prometheus HA replication works; the challenges it poses for long-term storage systems; our original solution (and the problems we encountered); and how we built a new system to address those issues and deliver a more robust, resilient, and consequently more useful, observability data platform.We’ll conclude with a brief overview of common issues beyond HA that are critical to think through and address in any production observability data system.How Prometheus HA worksAn architecture diagram illustrating two Prometheus High Availability (HA) clusters, with two Prometheus replicas each. Arrows from replicas to servers indicate scraping.Any production system that relies on Prometheus data for operations should deploy Prometheus with HA. When things go wrong, and your cluster is under stress, it’s vital that your observability systems keep running, so that you can diagnose and fix the problem.As a result, Prometheus users often deploy Prometheus asHigh Availability (HA) clustersto protect against any single Prometheus server crashing.Theseclustershave multiple identical Prometheus servers that run asreplicason different machines/containers. These servers scrape the same targets, and thus have very similar data (scrapes happen at slightly different times, which may lead to minor differences), so if one Prometheus server goes down, the other one(s) will have the same data.Thus, you can think of aPrometheus HA clusteras agroup of similar Prometheus servers that all scrape the same targets. Also, you can considerPrometheus replicasas the individual Prometheus servers within that group.The problem we - and all long-term storage systems - must solveSince different replicas within the same cluster contain essentially the same data, many users prefer to store long-term data from only one of the replicas to save on storage costs.However, in order to maintain high-availability properties, the long-term storage system must “deduplicate” data. This “deduplication” must also be resilient to the failure of any single Prometheus replica. Yet, because Prometheus expects data that is mostly regular (i.e., occuring at equally-spaced times), you also don’t want to switch between the replicas too often.Therefore, the basic solution elects a single replica to write data, and switches to another replicaonlyif the leader replica dies.The flow of data then looks something like:An example sequence of leadership switches between Replica 1 and Replica 2. Replica 2 takes over only when Replica one dies. Only the leader’s data is saved by Promscale.Our original solution (Using locks)Since the implementation challenge described above sounds a lot likeleader election, where we choose one replica out of many to be the “leader,” our original implementation reused PostgreSQL to implement leader-election using exclusive advisory locks. (Again, Promscale is built on top of TimescaleDB and PostgreSQL.)This is how our original solution worked:We coupled every Prometheus server with a unique Promscale connector.We asked all of the Promscale connectors to try to grab an exclusive database lock.Whichever Promscale connector “won” (and was able to grab the lock) became the leader.The Prometheus server that was paired with the Promscale leader (as determined in the prior step) became the Prometheus leader and had its data written to the database.A basic architecture for High Availability via leader election using locks.Flaws in our original designBut, as users started deploying Promscale, we discovered two main problems with our original implementation:Tight coupling between Prometheus and PromscaleChallenges handling delayed dataTight coupling between Prometheus and PromscaleThe first problem was the one-to-one mapping (i.e., coupling) between Prometheus and the Promscale connector. This made it impossible to independently scale the Prometheus layer from the Promscale layer and thus created scaling issues, hot spots, and inefficient resource utilization.In terms of scaling, this meant we could no longer provide multiple Promscale connectors to handle ingestion from one Prometheus server, necessitating large Promscale machines for handling load from large Prometheus servers.Having only one leader Promscale connector be able to write data at any one time also created hot spots and wasted resources: The Promscale connector that was the leader did most of the work (actually writing data to the database) while the non-leaders did almost nothing.This coupling also made deployments more complex, as Prometheus and Promscale - two seemingly independent services - had to be deployed in a one-to-one mapping. This wasespeciallycomplex in the Kubernetes ecosystem, where users had to inject Promscale as a side-car into the Prometheus pod.Challenges handling delayed dataThe second problem was processing delays, since the “who is the leader?” decision depended on wall-clock time and not data-time. If one of the Prometheus servers had an intermittent failure and lost its leader status, then came back up and tried to re-send data, that data would be considered “late” and be thrown away.For example, if replica A was the leader and crashed, replica B took over after some period of time. Now, imagine replica A comes back online and proceeds to send data from the period between A crashing and B taking over (e.g., if replica A didn’t finish sending all of the data it collected before it crashed, and then attempted to finish sending after coming back online). We couldn’t use that data to “fill in the gap” because B is now the leader, and all data from A had to be discarded.An improved solution (Using labels and leases)Our requirements for better solutionBased on what we saw with our initial design, we wanted an improved system that:Allowed multiple Promscale connectors to process write traffic at the same time to avoid hot spots and wasted resources.Allowed Promscale servers to be scaled and sized independently from Prometheus connectors.Spread work evenly across Promscale connectors.  In other words, we wanted to be able to put a load-balancer between the two systems.Base leadership decisions on data-time and not wall-clock-time, allowing data to be ingested from the correct leader (for the time-period) even if it arrives late.This led to the realization that what we really wanted was a system that allowed multiple Promscale connectors to write data coming from a single “leader” Prometheus server at the same time. In other words, we couldn’t elect a leader Promscale connector corresponding to a Prometheus server, rather we had to elect a leader Prometheus server directly and allow multiple Promscale connectors to process the data coming from that leader.Using immutable leases to elect a single leader Prometheus replicaAs we outlined above,  our original leader election system allowed only a single Promscale connector to obtain a lock and write data. But, this method could not be used to pick an identifier for a Prometheus leader. We needed to change the system to elect a single Prometheus leader replica – and share this information across multiple Promscale connectors.To do this, we useimmutable leaseswritten as records into a database table, stored within the same TimescaleDB instance as the Prometheus metric data itself.Here is how that works: For any given time period, we insert a lease record that maps the time period to a unique replica leader (e.g., replica A). This mapping is immutable: once it is created, no other replica can create another entry for the same time period in our lease record table. However, the time period is limited in time (e.g., 60 seconds). But as long as the most recent leader is still alive, it can keep expanding that time range forward in time (i.e., in 60-second increments).For example: if replica A becomes the leader for 00:00-00:60, at 00:30, it can extend its lease to be 00:00-00:90 (i.e., an additional 60 seconds, the maximum time range.). (Technically, the replica does not create and extend its lease directly, but rather a Promscale connector creates and extends the lease on the replica’s behalf.)If the leader dies (i.e., stops actively expanding the lease), another replica can become the leader for asubsequenttime period. To continue with the above example, if replica A dies at 00:75, replica C can create a new lease for 00:91-00:151, since replica A’s lead doesn’t expire until 00:90. In this example, we lose 15 seconds of data (data from 00:75-:00:90) – and users can mitigate any potential loss by setting a conservative lease timeout period, so only a few scrapes are lost. (This seems like a fair compromise to us. Also, to our knowledge, this kind of data loss happens with every single Prometheus long-term storage system that uses leader-election vs. retaining data from all replicas.)An example sequence of lease records indicating which replica was the leader at a particular point in time. Only the data coming from the leader (according to the data’s time) is saved by Promscale.The existence of records in the database allows multiple Promscale connectors to get a consistent view of the leases in the database. If multiple Promscale connectors try to create a lease at the same time (on behalf of different replicas), the database makes sure that there is only one winner, which is exposed consistently to all the other connectors.We actually maintain two tables: one access-optimized table that contains the current lease, and a second one that contains a log of all the leases taken for previous periods of time. We maintain this log to enable users to determine whether late-arriving data belonged to the leader, as well as for auditing or debugging purposes.An important detail here is that the lease is based on the time in the data, not wall-clock time.This makes the system safe for Prometheus servers that experience different latencies writing to the database, or that fall behind because of slow processing. In particular, if a slowdown or intermittent failure in one of the Prometheus servers causes “late” data, we use the lease log table to figure out which replica was the leader at the time corresponding to the “late” data. From there, we determine if the “late” data should be inserted, or dropped because it came from a non-leader.Figuring out which Prometheus replica sent the dataOnce we are able to pick a Prometheus replica leader, we still have a problem: how does a Promscale connector that can get data from multiple replicas know which data came from the leader (i.e., to save that data), and which came from the non-leaders (i.e., to discard that data)? To solve this problem, we took inspiration fromCortex’s idea to use  external labelsto signal the cluster and replica that’s associated with the data being sent from Prometheus.More concretely: since external labels are part of the HTTP request sent via Prometheus remote_write, we can now distribute requests to several Promscale connectors (e.g., via a load balancer) instead of having to maintain a 1:1 mapping – without losing the knowledge of which request came from which replica.Our new architectureThis new approach allows us to distribute write load among several Promscale connectors, so that our architecture now looks like this:Promscale’s new HA approach using leases and labels.This new architecture has several advantages:Multiple Promscale connectors can process write traffic concurrently, increasing write throughput.Promscale connectors can be sized and scaled independently of Prometheus servers.Traffic from Prometheus servers can be fairly distributed to Promscale connectors with a load balancer.Delayed data is now correctly handled, by saving it if it came from the “leader” replica for that data-time, or by discarding it otherwise.Edge-case: handling slowdownsWhile developing leader election systems many interesting edge cases show up.We thought we’d share one such edge-case we found surprising and didn’t expect: What if the “leader” Prometheus server sends data that is older than the data being sent by anon-leader? In this (slow) tortoise and (fast) hare scenario what should we do?We saw two options:Switch leader to the hare, and possibly introduce a lot of leader changes if this scenario occurs frequently.Stay with the tortoise and assume it will catch up over time.An additional confounding factor: how do we tell the difference between the tortoise being slow and just not working?In thinking through this scenario, we realized that Prometheus already has methods to adjust to slowdowns. For example, Prometheus could increase the number of queues it uses to send data via remote write, or it could slow down its scrape loop (and does this by default). It seemed like doing our own adaptation (option 1 above) ran a very real risk of creating chaos.Thus, we opted for option 2, trusting Prometheus to do the adaptation. We stay with the existing leader, unless we don’t receive new data from it in 30 seconds (of wall-clock time) and there is another Prometheus server sending data newer than the existing lease.Beyond high-availability Prometheus: a production-ready system for storing and collecting observability dataHere we described how we - Promscale - evolved our approach for supporting Prometheus high-availability. But of course, truly resilient systems need much more.For example, truly resilient systems need to support high-availability ateverystep of the ingest pipeline, and to support backups and disaster recovery (e.g., and recovery from the ever-present “fat finger”).Promscale supports these requirements in various ways.For Promscale, the ingest pipeline consists of 3 components – Prometheus server, Promscale connector, and TimescaleDB – each of which supports high-availability. We have already discussed the Prometheus high-availability setup. Promscale is a stateless service and so can safely be run as a replicated service behind a load balancer. TimescaleDB itselfcan be configured with multiple types of high-availability deploymentswhich allow the user to maketradeoffs in read performanceappropriate for their system.Aside from high availability throughout the ingest pipeline, a resilient system should also have a good way of performing remote backups. TimescaleDB supportscontinuous streaming backupsand point-in-time recovery. This protects you against data-center disasters as well as operational mistakes and fat-finger errors (and who hasn’t done /that/).Proper permissions and limited roles is another way to protect your data. That’s whyPromscale implements a role-based access control (RBAC)permission system at the database level. This allows you to grant separate permissions for reading data, writing data, and administering the database out of the box. Using PostgreSQL’s advanced permission systems, finer control is also possible.ConclusionIn summary, we’re excited about our new approach for ingesting data from Prometheus servers running in high-availability mode. We highlighted the importance of building systems where independent layers are loosely coupled and can scale independently – as well as why observability solutions must carefully consider the difference between wall-clock and data times (something we learned the hard way).We hope that the way we’ve drawn inspiration from other projects - specifically Cortex in this instance - serves as inspiration for others who wish to improve the Prometheus ecosystem. Promscale is 100% open-source, and we’re happy to work with anyone who’d like to adapt pieces of our design for their own projects (and continue the virtuous cycle 🔥).We’ll continue to invest in future improvements, both in how we support Prometheus high availability and the overall Promscale platform. We also welcome any feedback or direct contributions from community members.Get started and join the Promscale communityTo get started with Promscale:Install Promscale today via Helm Charts, Docker, and others.See GitHub for more information(GitHub ⭐️ welcome and appreciated! 🙏.)If you are looking for managed Prometheus storage, get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.Check out ourGetting Started with Promscale tutorialfor more on how Promscale works, installation instructions, sample PromQL and SQL queries, and more.WatchPromscale 101 YouTube playlistfor step-by-step demos and best practice.To get involved with the Promscale community:Reach out inTimescale Slack(#prometheus)for technical help or questions.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/promscale-0-4-drawing-inspiration-from-cortex-to-rewrite-support-for-prometheus-high-availability/
2022-05-18T13:12:31.000Z,Get Down(sampled) With Prometheus Recording Rules in Promscale,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.This is KubeCon Europe week, and at Timescale, we’re celebrating it the way we know best: with a new edition of #AlwaysBeLaunching! 🐯 🎊 During this week, we are releasing new features and content every day, all focused on Promscale and observability. Do not miss any of it by following our hashtag #PromscaleKubecon on Twitter—and if you’re at KubeCon, come visit us at booth G3! We have awesome demos (and swag) to give you!One more day, one more launch! Yesterday, we announced the support forPrometheus Alerting Rules in Promscale—and today, it’s the Recording Rules’ turn! 🔥In Prometheus,Recording Rulesallow you to periodically write the output of PromQL expressions that are frequently used or computationally expensive as a new time series, so you can conveniently query them later while getting faster response times.Recording Rules are very useful as a downsampling mechanism, as you can use them to roll up metrics into larger time periods. By directly configuring Recording Rules in Promscale, you candefine a retention policy that actually drops the source data after a predefined amount of time, which is not possible to do directly with a single Prometheus instance which has a single retention time for all metrics.But Recording Rules are not the only way to downsample data in Promscale. Since Promscale is built on top of PostgreSQL and TimescaleDB, you can both usePrometheus’ Recording Rules written in PromQLandTimescaleDB’s continuous aggregates written in SQLfor downsampling. This gives you maximum flexibility to choose the method and query language that better adapts to the particular metric you’d like to downsample. For more information on the differences between both methods and when to use one or the other,check out this blog post we published on the topic.To learn how you can configure recording rules in Promscale, keep reading. And if you want to try out Promscale, you can install it for freehere.✨ Promscale is the unified observability backend for metrics and traces built on PostgreSQL and TimescaleDB. It brings you observability powered by SQL without losing native support for Prometheus and PromQL; you can store your Prometheus metrics and your OpenTelemetry traces in Promscale, using it together with Grafana and Jaeger for an integrated observability experience. If you’re new to Promscale, check out ourwebsiteanddocumentationto learn more.Configuring Recording Rules in PromscalePrometheusRecording Rulesallow the creation of new metrics from existing metrics using PromQL, usually because we know we are going to use them frequently or they run too slowly to generate at view time. For example, if we know that we are always going to sum our requests by HTTP response code, it might be worth looking at a Recording Rule, especially if we have a very large number of instances.A basic recording rule is expressed as a record (which defines the name of the new time series) and an expression (which defines the query to run).Promscale supports the full PromQL language, so any rule which Prometheus can evaluate can also be used in Promscale on similar metric data to create new time series with Recording Rules.Recording Rules are configured in Promscale similarly to in Prometheus—using a YAML file that points at YAML files containing recording rules to load. In fact, both file formats are identical, so if you have a working Prometheus configuration that includes arule_filesblock and some recording rules, you can point Promscale at the config, and things will work.So what does that config look like? A basic example of the config needed would be aprometheus.ymlwith the following content:# Rules and alerts are read from the specified file(s)
rule_files:
  - rules.ymlRules will be read from therules.ymlwhich contains a list of PromQL rules (or alerts) to load and evaluate. An example of the file with a recording rule that pre-calculates the rate of CPU usage would look like this:groups:
- name: rules
  rules:
  - record: instance_cpu:node_cpu_seconds_not_idle:rate5m
    expr: >
      sum(rate(node_cpu_seconds_total{mode!=""idle""}[5m])) 
      without (mode,cpu)Once we have the config files, Promscale can be started with either themetrics.rules.prometheus-configor thePROMSCALE_METRICS_RULES_PROMETHEUS_CONFIGenvironment variable pointing at theprometheus.ymlfile.Check Out an ExampleIf you’d like to give this a go, then you can use the docker-compose file, which is located in thehttps://github.com/timescale/promscalerepository, to see a recording rule working.If you execute the following commands from a shell:git clone https://github.com/timescale/promscale
cd promscale/docker-compose
docker-compose upThen you should see the stack come up. After waiting for it to stabilize you can connect to the TimescaleDB database and see the data from the recording rule with the following command:docker exec -it docker-compose-db-1 psql -c \
  'SELECT * FROM ""instance_cpu:node_cpu_seconds_not_idle:rate5m""   
  LIMIT 10'You should see a few rows of metric data. (If you don’t, you may need to wait a little longer for the rule to evaluate.)Wrap UpPromscale now natively supports bothalerting rulesandrecording rules, giving users the option to not run a full Prometheus instance for collecting metrics (saving valuable machine resources) while still maintaining access to these great features.If you have any questions about this new functionality, please reach out to us—we’ll be more than happy to help! You can find us in ourCommunity Slack(make sure to join the #promscale channel). And for hands-on technical questions, you can also post in theTimescale Forum.#AlwaysBeLaunching!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/get-down-sampled-with-prometheus-recording-rules-in-promscale/
2022-10-26T13:12:23.000Z,"Using PostgreSQL as a Scalable, Durable, and Reliable Storage for Jaeger Tracing","⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Jaegeris an open-source solution for troubleshooting cloud-native applications using distributed tracing. It’s aCloud Native Computing Foundation (CNCF) project at the Graduated maturity levelthat is widely adopted and commonly used alongsidePrometheus to monitor applicationsrunning in Kubernetes.Jaeger supports a growing number of storage backends and has two built-in ones: in-memory storage and local disk storage based on BadgerDB. They were designed to be used during testing but not for production use, where you need to ensure reliability, scalability, and durability, so your data survives restarts and system crashes.To solve that problem, Jaeger has out-of-the-box support for storing traces in Elasticsearch and Cassandra. Elasticsearch is currently the recommended storage. Additionally, Jaeger offers agRPCplugin API that you can use to integrate other storage systems.In our conversations with Jaeger users, we learned that they often go with one of the storages not recommended for production (in-memory or BadgerDB). This is due to their lack of operational experience with Elasticsearch and Cassandra and their general awareness of how complex it is to run and manage these storage systems.On the other hand, PostgreSQL is a familiar database for many engineers who use it to back their applications. It is also well-known for its friendly developer experience—unsurprisingly, it was voted the most loved database inStack Overflow’s Developer Survey 2022.Additionally, PostgreSQL offers a rich query experience with SQL, allowing engineers to analyze the trace data more deeply than when using Elasticsearch. There is a reason why SQL is the universal language for analytics.This is why today we are announcing the ability to use PostgreSQL as a certified, durable, reliable, and scalable storage for Jaeger with Promscale.Promscale is a unified metric and trace storage for Prometheus, Jaeger, and OpenTelemetry built on PostgreSQL and TimescaleDB. With Promscale, you get a centralized and reliable long-term storage for your metrics and traces that offers the following:Full Jaeger support: passes all the Jaeger storage certification tests, has native support for OpenTelemetry, and can be used as the metric storage backend for Jaeger’s Service Performance Management (SPM) feature.First-class Prometheus support: 100 % PromQL-compliant and support for PromQL alerts and recording rules, exemplars, Prometheus high availability, and multi-tenancy.Flexible storage:configurable downsampling and retention policies, including per-metric retention, data backfilling and deletion, and full support for both PromQL and SQL.Rock-solid foundation: built on the maturity of PostgreSQL and TimescaleDB with millions of instances worldwide. A trusted system offering scalability, high availability, replication, and data integrity.Want to try it out? The easiest way to get started is tosign up for Timescale (create a free 30-day account, no credit card required).Self-hostingis also available for free.Jaeger Tracing With PostgreSQL: How This WorksJaeger’s architecture is made of five components:Client libraries: these can belong to Jaeger or OpenTelemetry.The Agent: batches trace spans sent by client libraries and sends them to the Collector.The Collector: receives all the trace spans, validates them, transforms them if needed, and sends them to the storage.The Storage Plug-in: required for storage backends that are not supported out-of-the-box by Jaeger.The Storage: where trace spans are saved and from where they are retrieved for display in the UI.The Query Service: translates UI requests into backend queries.The UI: displays the trace data.SourceIntroducing support for a new database requires a new storage plugin. There are two ways to create it: through a binary that first needs to be packaged with Jaeger and then deployed or a remote storage plugin running on the storage side that can be enabled in Jaeger with a simple configuration change. The latter is an improvement that the Promscale teamcontributed to Jaegerlast year. In both cases, the same storage gRPC API needs to be implemented so that Jaeger can write and read data from the database.Promscale implements the remote storage plugin model, making integration with Jaeger much easier. It automatically creates and manages the schema in the database and converts the Jaeger data into that schema. The only change required in Jaeger to start storing and querying traces in PostgreSQL is to change a configuration parameter.Additionally, we use TimescaleDB on top of PostgreSQL to improve ingest and query performance and reduce storage requirements, courtesy ofTimescaleDB’s columnarcompression. Our initial tests show ingestion rates above 100,000 spans per second on a single database node (16 CPU, 64 GB) with more than 90 % data compression on disk. We’ll publish detailed performance test results in the future.The certification processUp until recently, the only storage backends that the Jaeger project promoted were Elasticsearch and Cassandra (for production use) and in-memory and BadgerDB (for testing). Why? Because Jaeger could not ensure that others would work well.There is a long list of storage plugins that were built to integrate Jaeger with other databases. Unfortunately, the Jaeger community had difficulties assessing the quality of those plugins (do they support all Jaeger features, or are there any limitations?) and how well-supported they are (do they work with recent versions?). It’s also tough to find them since Jaeger does not promote them.To address this issue, we collaborated with the Jaeger maintainers (special thanks to Yuri Shkuro, the project’s creator) to come up with acertification processfor Jaeger’s storage backends. The goals of the process are two-fold:Provide an easy way for any storage backend to measure and prove its compliance against Jaeger.Make it easier for the Jaeger community to discover storage backends that are certified for 100 % compliance.The end result is a Go package that can be easily used to run all of Jaeger’s gRPC storage tests against any storage:import (
	jaeger_integration_tests ""github.com/jaegertracing/jaeger/plugin/storage/integration""
)

func TestJaegerStorageIntegration(t *testing.T) {
        ...
	si := jaeger_integration_tests.StorageIntegration{
		SpanReader: createSpanReader(),
		SpanWriter: createSpanWriter(),
		CleanUp: func() error { ... },
		Refresh: func() error { ... },
		SkipList: []string {  // Skip any unsupported tests
		},
	}
	// Runs all storage integration tests.
	si.IntegrationTestAll(t)
}This is an exampleof how we integrated these tests into the automated test suite used by Promscale.The storage backend needs to make the results publicly available to claim certification. Ideally, the tests are integrated into the continuous integration pipeline, and the results are published with every new version. As an example, this is an excerpt of the Promscale results (full test resultshere):=== RUN   TestJaegerStorageIntegration/streaming/GetDependencies
    integration.go:370: Skipping GetDependencies test because dependency reader or writer is nil
--- PASS: TestJaegerStorageIntegration (19.41s)
    --- PASS: TestJaegerStorageIntegration/sequential (9.33s)
        --- PASS: TestJaegerStorageIntegration/sequential/GetServices (0.27s)
        --- PASS: TestJaegerStorageIntegration/sequential/GetOperations (0.04s)
        --- PASS: TestJaegerStorageIntegration/sequential/GetTrace (0.17s)
            --- PASS: TestJaegerStorageIntegration/sequential/GetTrace/NotFound_error (0.00s)
        --- PASS: TestJaegerStorageIntegration/sequential/GetLargeSpans (6.05s)
        --- PASS: TestJaegerStorageIntegration/sequential/FindTraces (0.62s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_in_one_spot_-_Tags (0.24s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_in_one_spot_-_Logs (0.02s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_in_one_spot_-_Process (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_in_different_spots (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Trace_spans_over_multiple_indices (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Operation_name (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Operation_name_+_max_Duration (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Operation_name_+_Duration_range (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Duration_range (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/max_Duration (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/default (0.01s)
 --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_+_Operation_name (0.02s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_+_Operation_name_+_max_Duration (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_+_Operation_name_+_Duration_range (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_+_Duration_range (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Tags_+_max_Duration (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Multi-spot_Tags_+_Operation_name (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Multi-spot_Tags_+_Operation_name_+_max_Duration (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Multi-spot_Tags_+_Operation_name_+_Duration_range (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Multi-spot_Tags_+_Duration_range (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Multi-spot_Tags_+_max_Duration (0.01s)
            --- PASS: TestJaegerStorageIntegration/sequential/FindTraces/Multiple_Traces (0.01s)You may notice in the full output that two tests are skipped:--- SKIP: TestJaegerStorageIntegration/sequential/GetDependencies (0.00s)
…
--- SKIP: TestJaegerStorageIntegration/streaming/GetDependencies (0.00s)The reason why they are skipped is that the Jaeger remote storage plugin interfacedoes not support the ability to write dependenciesyet. This data feeds the System Architecture tab in Jaeger.As of today, Promscale is one of only two external certified storage backends.Setting Up Jaeger With Promscale and PostgreSQL as a Storage BackendThe complete high-level architecture of Promscale looks as follows:We will only focus on the Jaeger-Promscale integration paths via the remote storage API for ingesting and visualizing trace data.To integrate Jaeger with Promscale, you need to perform the following steps:Set up the database consisting of PostgreSQL with TimescaleDB and the Promscale extension.Deploy the Promscale Connector linked to the database.Configure Jaeger to use PostgreSQL via Promscale as a storage backend.1. Set up the databaseThe easiest way to get the database up is to use Timescale:Sign upfor a free 30-day trial. No credit card required.Create a serviceusing the advanced configuration. A single node with 2 CPUs / 8 GB supports up to 10,000 spans per second which will be plenty for testing. Also, increase storage to 500 GB. If you have specific requirements, you can check ourresource recommendation guide. Copy the Service URL on the page after clicking on Create service. We’ll need it later.If you prefer to self-host, follow the instructions inthe documentationfor your specific environment.2. Deploy the Promscale ConnectorThe Promscale Connector is deployed in your infrastructure.Deploy on Kubernetes with HelmReplace<service-url>with the one you copied when you set up the database in the instructions below:helm repo add timescale 'https://charts.timescale.com'
helm repo update
helm install promscale timescale/promscale --set connection.uri=<service-url>Deploy with DockerReplace<service-url>with the one you copied from Timescale in the instructions below:docker run --name promscale -d -p 9201:9201 -p 9202:9202 -db.uri=<service-url>For other deployment options, seethe documentation.3. Configure JaegerTo set up Jaeger to connect to a storage via gRPC, you need to set two parameters:span-storage.type, which defines the type of storage Jaeger should use. This needs to be set to grpc-plugin to instruct Jaeger to use a storage that is connected via a gRPC plugin.grpc-storage.server, the server implementing the remote storage gRPC API or, in our case, the Promscale Connector. This should be set to<promscale-host>:9202. Replace<promscale-host>with the actual hostname or IP where you’ve deployed the Promscale Connector.These parameters have to be passed to the Jaeger Collector and the Jaeger Query components (or to Jaeger all-in-one if that is what you are using). They can be passed as arguments to the CLI, configuration files, or environment variables. In the latter case, you have to replace the “-” and “.” in the parameter names with “_” and convert all characters to uppercase.For example, to run the Jaeger Collector with Docker using Promscale as a storage backend, you would run:docker run \
  -e SPAN_STORAGE_TYPE=grpc-plugin \
  -e GRPC_STORAGE_SERVER=""<promscale-host>:9202"" \
  jaegertracing/jaeger-collector:1.38Additional Benefits of Using Promscale as a Jaeger Storage BackendAs we mentioned, Jaeger comes with two embedded storages (in-memory and BadgerDB), but they are not designed to be used in production because they aren’t built to be durable, reliable, and scalable.For production workloads, Jaeger recommends Elasticsearch as the main option. Unfortunately, many engineers are not very familiar with Elasticsearch, and operating it is known to be pretty complex. As a result, many Jaeger users rely on in-memory or BadgerDB with the risk of losing access to their trace data when they most need it in the middle of a production incident.PostgreSQL is a much more widely used database backing millions of applications; therefore, many more engineers are familiar with setting it up and operating it. On top of that, PostgreSQL is well-known for being very developer friendly. The ability to use PostgreSQL as a certified storage backend for Jaeger makes it easier for the community to adopt a durable, reliable, and scalable database to store their production traces. This ensures they always have access to their traces when they most need them.It doesn’t stop there, though. There are two great additional benefits that you’ll get for free:A storage backend for the Jaeger Service Performance Management (SPM) feature.Jaeger SPM takes Jaeger from a tool that troubleshoots problems by exploring traces to a tool that allows proactive monitoring of your applications by looking at how your different services perform over time.SQL query capabilities on top of your traces so you can build dashboards in Grafana to get new insights from your traces. Get started with ourout-of-the-box dashboards.We have talked about querying traces with SQL several times in the past.We showed you how you could get more value out of traces by using this query languageandother observability and data visualization toolsand helped youunderstand your traces like never before using OpenTelemetry and PostgreSQL. Stay tuned, as we will publish a blog post about using Jaeger SPM with Promscale very shortly.Become a JaegerMaster With PostgreSQL and PromscaleUntil recently, engineers who wanted to use a database that was 100 % Jaeger-compliant had to resort to Elasticsearch or Cassandra, both known for being difficult to operate.Working alongside the maintainers of the Jaeger project, we created a new certification process for storage backends so that the community can use other databases to store their traces confidently.Promscale is one of the only two currently certified Jaeger storage backends. With Promscale, the Jaeger community can finally make the most of a well-known, widely available, and user-friendly database to store their traces: PostgreSQL. You can connect Jaeger to Promscale with a simple configuration change, unlocking new capabilities, like Jaeger’s Service Performance Management and SQL queries, to analyze traces and build Grafana dashboards.Get started now withPromscale on Timescale (free 30-day trial, no credit card required)orself-hostfor free.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/using-postgresql-as-a-scalable-durable-and-reliable-storage-for-jaeger-tracing/
2022-05-20T13:51:57.000Z,Kubernetes Observability in One Command: How to Generate and Store OpenTelemetry Traces Automatically,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.OpenTelemetry traces hold a treasure trove of information to understand and troubleshoot distributed systems—but your services must be first instrumented to emit OpenTelemetry traces to realize that value. Then, those traces need to be sent to an observability backend that allows you to get answers to arbitrary questions on that data. Observability is an analytics problem.Earlier this week, we partly solved this question by announcingthe general availability of OpenTelemetry tracing support in Promscale, bringing observability powered by SQL to all developers. With the addition of full support for SQL, the lingua franca of analytics, to interrogate your trace data, we handled the analysis problem. But we still need to tackle the first part: instrumentation.To get your services to emit trace data, you have to add OpenTelemetry instrumentation to their code manually. And you have to do it for all your services, and all the frameworks you use or you won’t get visibility into the execution of each request. You also need to deploy the OpenTelemetry Collector to receive all the new traces, process them, batch them, and finally send them to your observability backend. That’s a lot of time and effort.What if you didn’t have to do all that manual work and could get up and running in minutes instead of hours or even days? What if you could also set up an entire observability stack and connect all the components automatically? And what if I told you that you could do all of this with a single command?I’m not crazy. I’m just atobsuser 😎Tobs, the observability stack for Kubernetes, is a tool you can use todeploy a complete observability stack in your Kubernetes cluster in a few minutes. The stack includes the OpenTelemetry Operator, the OpenTelemetry Collector, Promscale, and Grafana. It also deploys several other tools, like Prometheus, to collect metrics from the Kubernetes cluster and send them to Promscale. Andwith our latest release, tobs includes support to automatically instrument your Python, Java, and Node.js services with OpenTelemetry traces via the OpenTelemetry Operator.Yes, you read that correctly:automatic! You don’t need to change a single line of code in your services to get them instrumented. And the icing on the cake? You can deploy everything by executingonehelm command.With tobs, you can install your observability stackandtake care of the first level of your OpenTelemetry instrumentation in just a few steps. Say farewell to tedious configuration work as your frameworks instrument themselves.If you want to learn how to do it, keep reading this blog post. First, we will explain how everything works, dissecting what the OpenTelemetry Operator really does under the hood. Next, we’ll demonstrate how you can put this directly into practice with an example:We will install a complete observability stack in our Kubernetes cluster through tobs.We will deploy a cloud-native Python application.We’ll check how our app has been automatically instrumented with OpenTelemetry traces, thanks to the magic tricks 🪄 performed by tobs and the OpenTelemetry Operator.The OpenTelemetry OperatorOpenTelemetryis an open-source framework that can capture, transform and route all types of signals (traces, logs, and metrics). In most cases, you’d use theOpenTelemetry SDKto generate the signals in your application code. But, in some cases, OpenTelemetry can instrument your code automatically—i.e., when your application framework is supported and when you’re using a language OpenTelemetry can inject code into. In this case, your systems will start generating telemetry with no manual work needed from you.To understand how OpenTelemetry does that, we first need to get familiar with theOpenTelemetry Operator. The OpenTelemetry Operator is an application that implements theKubernetes Operator patternto interact with two CustomResourceDefinitions (CRDs) in a Kubernetes cluster.Diagram illustrating how the OpenTelemetry operator interacts with Kubernetes✨ One of the Promscale Team members, Vineeth Pothulapati (@VineethReddy02 in GitHub), is a maintainer of the OpenTelemetry Operator. Cheers to Vineeth!Based on changes inCustomResourceDefinitions (CRD)instances, the Operator manages two things for us:Creating and removing OpenTelemetry Collector instancesInjecting the libraries and binaries needed by OpenTelemetry auto-instrumentation directly into your podsLet’s unpack these two tasks in more detail.Managing the OpenTelemetry CollectorThe first concern of the OpenTelemetry Operator is to deployOpenTelemetry Collectorinstances. These instances will be used to route signals from their source (your workload and Kubernetes itself) to their intended destination (a storage system that supports the OpenTelemetry Protocol or another Collector outside your cluster).Collectors can be deployed in three different ways:As a Kubernetes Deployment:this is the default option, which allows the Collector to move between nodes as needed, supporting scalability up and down.As a Kubernetes Daemonset:this option will deploy one Collector per node, and it can be useful when you want to make sure your signals are processed without any network overhead.As a Sidecar:which is injected into any new annotated pods (usingsidecar.opentelemetry.io/inject: true). This can be great when a Collector needs the specific config of a pod (e.g., maybe it needs some dedicated transformations).You can mix and match these Collector patterns if you want. For example, you could set up a sidecar to do some transformations for the pods in a deployment and then send them to a global Collector which is shared with your other workloads.The configuration which defines these Collector instances is modeled in the Collector CRD (opentelemetrycollectors.opentelemetry.io). Multiple instances are allowed to achieve more complex patterns. The deployment type is selected with themodesetting, accompanied by a raw config string that is passed to the Controller verbatim, and loaded as the configuration. An example of a CRD which creates an Operator using the Deployment pattern would be:apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: tobs-tobs-opentelemetry
  namespace: default
Spec:
  mode: deployment
  config: |
    receivers:
      jaeger:
        protocols:
          grpc:
          thrift_http:

      otlp:
        protocols:
          grpc:
          http:

    exporters:
      logging:
      otlp:
        endpoint: ""tobs-promscale-connector.default.svc:9202""
        compression: none
        tls:
          insecure: true
      prometheusremotewrite:
        endpoint: ""tobs-promscale-connector.default.svc:9201/write""
        tls:
          insecure: true

    processors:
      batch:

    service:
      pipelines:
        traces:
          receivers: [jaeger, otlp]
          exporters: [logging, otlp]
          processors: [batch]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [prometheusremotewrite]As we’ll see in the example later, when you usetobs, you don’t need to worry about all these configuration details. One of the great things about tobs is that it will install a Collector for you, which will directly send data to a local Promscale instance.Adding OpenTelemetry Auto-Instrumentation to KubernetesThe second concern of the Operator is to inject the libraries and binaries needed for OpenTelemetry auto-instrumentation into pods. For this to work, these pods need to hold Java, Python, or Node.js applications (OpenTelemetry will support more languages in the future).The Kubernetes manifest file used to deploy those pods must include an annotation to instruct the OpenTelemetry Operator to instrument them:instrumentation.opentelemetry.io/inject-<language>: ""true""Wherelanguagecan bepython,java, ornodejs.When the annotated pods start aninitcontainer is created, injecting the needed code and altering the way the pod runs its code, using the correct OpenTelemetry auto-instrumentation method. Practically speaking, this means that without any code changes, you can get the benefit of auto-instrumentation when using Kubernetes. The config also defines the OpenTelemetry Collector endpoint to which these traces will be sent, the types of information propagated, and the method (if any) we use to sample the traces. (Forfull details on the CRDs, see the documentation).An example of a custom resource to auto-instrument Python, Java, and Node.js apps would look like this:apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: tobs-auto-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://tobs-opentelemetry-collector.default.svc:4318 
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    argument: ""0.25""
    type: parentbased_traceidratioOnce again, if you are usingtobs, you won’t need to create these custom resources yourself. Tobs will ensure that the cluster is automatically configured to instrument any annotated pods without any action required from you. All you need to do is add one of the following annotations to the pods which you want to collect traces from:instrumentation.opentelemetry.io/inject-java: ""true""
instrumentation.opentelemetry.io/inject-nodejs: ""true""
instrumentation.opentelemetry.io/inject-python: ""true""Let’s see how this works in practice with an example.Using the OpenTelemetry Operator and TobsIn this section, we’ll use ourmicroservices demo application, which consists of an over-engineered password generator app.In the repo,you can find both an instrumented version and an uninstrumented version, which is the one we’ll be using for this example.To run this, you will first need a working Kubernetes cluster with cert-manager installed, access viakubectl(at least version 1.21.0 will be needed) configured, andhelminstalled. To deploy and run all the different components, you will need about 4 CPU cores and 8 GB of RAM available in your Kubernetes cluster.If you don’t have cert-manager in your cluster, you will need to install it by using this command:kubectl apply -f 
https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yamlOnce you are ready, let’s use the Timescale helm chart to install tobs. Run the following commands from the command prompt:helm repo add timescale https://charts.timescale.com/ --force-update
helm install --wait --timeout 10m tobs timescale/tobsTobs will take a few minutes to install, but eventually, you will see an output similar to this:#helm install --wait tobs timescale/tobs
NAME: tobs
LAST DEPLOYED: Thu May 19 11:22:19 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
###############################################################################
👋🏽 Welcome to tobs, The Observability Stack for Kubernetes

✨ Auto-configured and deployed:
🔥 Kube-Prometheus
🐯 TimescaleDB
🤝 Promscale
🧐 PromLens
📈 Grafana
🚀 OpenTelemetry
🎯 Jaeger

###################################👉 Troubleshooting tip:If you get this error messageINSTALLATION FAILED: rate: Wait(n=1) would exceed context deadline, this most likely indicates that there are not enough resources available in your cluster.Once tobs' installation is completed, check your Kubernetes cluster to confirm all components have been deployed correctly:kubectl get podes --all-namespaces | grep ""tobs-""👉Troubleshooting tip:If some pods are in pending or error state, you can usekubectl describe pod <pod-name>orkubectl logs <pod-name>to understand what may be the problem.Now, we can import the uninstrumented Kubernetes microservices from the OpenTelemetry Demo GitHub repo (https://github.com/timescale/opentelemetry-demo).If you review the code in theuninstrumentedfolder, you will see that it makes no mention of OpenTelemetry. For example, take a look at the Python file for theloadmicroservice (this service drives traffic to the other services by making password requests):#!/usr/bin/env python3
import requests

def main():
    while True:
        try:
            response = requests.get('http://generator:5000/')
            password = response.json()['password']
            print(password)
        except Exception as e:
            print('FAILED to get a password!')


if __name__ == '__main__':
    main()By importing these microservices into a cluster with tobs installed, they will get automatically instrumented with OpenTelemetry traces.To bring up the demo app, run:kubectl apply -k 'http://github.com/timescale/opentelemetry-demo/yaml/app'When the process finishes and the application is deployed, it will be already instrumented with OpenTelemetry tracing. Traces are now being generated and sent to Promscale automatically.How did this magic happen?Here’s a summarized explanation:Each pod is annotated withinstrumentation.opentelemetry.io/inject-python: ""true"", so as they start they are noticed by the OpenTelemetry Operator.Next, an init container is added using a mutating webhook, injecting the Python libraries and code needed to enable instrumentation.The trace data is then sent to the OpenTelemetry Collector noted in the Instrumentation CRD.The OpenTelemetry Collector sends the data to Promscale (and into TimescaleDB), from which it can be directly queried with SQL or accessed by tools like Grafana for visualization.Let’s take a look at our automatically-generated traces directly from Grafana (which tobs has also automatically installed in our cluster).To get the password of the admin user for the Grafana instance, run the following commands:kubectl get secret tobs-grafana -o jsonpath=""{.data.admin-password}"" | base64 -d 
kubectl port-forward svc/tobs-grafana 3000:80Then, navigate tohttp://localhost:3000/d/vBhEewLnkand use the password you just retrieved to log in as the admin user.ThePromscale application performance monitoring (APM) dashboardswill display, showing you insights about the demo app. tobs directly imports this set of out-of-the-box, production-ready dashboards, which we built in Grafana using SQL queries against the trace data that, in this case, it's being automatically generated by the demo microservices. The figure below shows one of the dashboards, ""Service Details"".Service Details dashboardpopulated with traces from the demo appFor more information on these pre-built dashboards,check out this blog post(navigate to the section “A Modern APM Experience Integrated Into Grafana”).We have got all this information with no instrumentation code in any of the Python services!ConclusionOpenTelemetry tracing has never been more accessible.If your microservices are written in one of the languages currently supported by the OpenTelemetry Operator, you can immediately start collecting and storing traces, with minimal manual work needed from your side.The only two steps you have to take are:Install tobs in your Kubernetes cluster via Helm. (Please, notice that you’ll have to install tobs using the Helm route for this latest release to work, not the CLI.)Add anannotationto the microservices pods you’d like to collect traces from (e.g.,instrumentation.opentelemetry.io/inject-python: ""true"") before deploying them.Your microservices will be automatically instrumented with OpenTelemetry traces, and your traces will be automatically stored in Promscale, the unified observability backend for metrics and traces built on PostgreSQL and TimescaleDB.You will immediately get insights into how your systems’ performance throughPromscale’s pre-built APM dashboards, and you will be able toquery your traces using SQL.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/generate-and-store-opentelemetry-traces-automatically/
2022-12-15T15:05:03.000Z,What Is Distributed Tracing and How Jaeger Tracing Is Solving Its Challenges,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.With the increased adoption of distributed systems, where the many system components are located on different machines, it has become both more important and more difficult to monitor these systems effectively. As more things can go wrong, quickly identifying and solving them can mean the difference between success and failure.That fine line has been driving the rapid adoption growth for popular technologies such as Jaeger and OpenTelemetry. These tools/libraries provide insight into how your application operates and performs through distributed tracing, allowing developers to see how the application requests are handled across their distributed systems. But more on that later.A popular tool for distributed tracing is Jaeger, which introduced a new experimental feature called Service Performance Monitoring earlier this year. As the name suggests, this feature adds performance monitoring to your traditional distributed tracing tool.This blog post will discuss what problem this feature solves and why it is a welcome addition to Jaeger. On top of that, we will take a deep look at how the OpenTelemetry Collector interacts with Jaeger to aggregate traces into Prometheus metrics and neatly graph them inside Jaeger’s UI.Lastly, we will walk you through a step-by-step tutorial on hitting the ground running using the OpenTelemetry Collector and Promscale (our scalable storage for Prometheus and Jaeger) to generate and store tracing and metrics data in the same time-series database!Let’s get started.What Is Distributed Tracing?First, let’s go back quickly to distributed tracing.Distributed tracingis the process of following and measuring requests through your system as they flow from microservice to microservice or frontend mobile application to your backend or from a microservice to a database.Spansare the data collected from distributed tracing inter-service requests and contain a wide range of information about the request. The group of spans that originates from the same initial request is called atrace.How Do Spans and Traces Work?Traces are propagated between services through metadata attached to the request, which we call context. When a request first enters a distributed system, your instrumentation (most likely OpenTelemetry) creates a root span. All subsequent requests between microservices will create a child span with a context that contains the following information:Its own IDThe ID of the root spanThe ID of its parent spanSome timestamps and other logsIf you start at the root span and follow the child ID of every span until you reach a leaf span (a span with no children), much like a binary tree, you can accurately reconstruct the services your request/trace has reached without having to collect the traces in order.Doing it this way suits a distributed system much better as the instrumentation doesn’t need to wait for confirmation that the parent span has been written to a persistent store, nor does it needs to know about spans being instrumented simultaneously.The Challenges of Distributed TracingUsually, you combine tracing with more traditional logging and metrics instrumentation. Still, due to the breadth and depth of traces, you can extrapolate a wide variety of helpful information from those traces. Each span (and therefore trace) starts with a timestamp from which you can calculate the total amount of time each request took.A well-instrumented tracing system ensures that if any event or error occurs, it gets stored in the context of said span. When following the spans from the root spans to the leaf spans, you can summate the number of errors (or lack thereof) within your request’s lifetime. When looking at all collected traces, you can calculate the total number of traces generated in a set timeframe and the percentage of erroneous requests.While this can provide an accurate view of what is going on inside your distributed system, it is computationally expensive to iterate through thousands (if not hundreds of thousands) of traces—each of which can contain hundreds of spans—to calculate a single metric.A dashboard with automatic refresh functionality will make it, but it will unnecessarily slow down your metrics store's insert and collection performance. Why? You will have to reread all traces within a certain time frame from your database and recalculate all metrics with the newly collected traces (a majority of which had already been calculated in the previous refresh).Another problem is that since trace data is produced in such huge volumes, it is commonplace to only instrument a set percentage of requests. This is called “sampling” and provides a way to balance the observability of a system and the computational expense of tracing. If our sampling rate is set at 15 percent, we only trace 15 out of 100 requests, for example.This is a problem for the accuracy of the metrics deducted from these traces, as it will skew our final results. And it is even more so when doing tail-based sampling, where we wait until the trace has been completed before sampling to decide to keep traces with high latency or where an error occurs. While unlikely, those 15 traces may contain an error.In that case, our dashboard would report a 100 percent error rate, even though the 85 other requests could be perfectly fine. Seeing inaccurate metrics during a crisis can lead to ahigher MTTRand more error-prone debugging, which is undesirable, to say the least!A great way to circumvent skewing our service performance metrics is by aggregating the metrics of a collection of traces after they have been completed and before sampling is applied. Doing it this way ensures an accurate result, even if we don’t keep all the traces.Another added benefit is that the processing happens as the traces are generated instead of when they are queried. This prevents your storage backend from overloading when you query the service performance metrics. Where the storage backend would have to read each span of each trace and aggregate it in real time, it can now just return the pre-aggregated metrics.What Is Jaeger's Service Performance Monitoring?Now, let’s see how Jaeger, a renowned tool for distributed tracing, is solving these challenges to provide accurate service performance metrics.Traditionally, Jaeger has only allowed storing, querying, and visualizing individual traces, which is great for troubleshooting a specific problem but not useful for getting a general sense of how well your services perform. That was until they introduced, earlier this year, a new experimental feature called Service Performance Monitoring (SPM).To use the SPM feature within the Jaeger UI, you are required to usethe OpenTelemetry Collector to collect tracesand a Prometheus-compatible backend. Traces enter the OpenTelemetry Collector at one of two trace receivers: OpenTelemetry or Jaeger, depending on the configuration.From there, they are sent to the configured processors. In our case, these are theSpanmetrics ProcessorandBatch Processor. The appropriate trace exporter exports the metrics (as usual) to a Jaeger- or OpenTelemetry-compatible trace storage backend.The Spanmetrics Processor calculates the metrics, and then you use the Prometheus or thePrometheusRemoteWrite exporterto get the metrics into a Prometheus-compatible backend.Below is an architecture diagram of the solution and how spans and metrics flow through the system.How the Spanmetrics Processor WorksThe Spanmetrics Processor receives spans and computes aggregated metrics from them.It creates four Prometheus metrics:calls_total:The calls_total metrics is a counter which counts the total number of spans per unique set of dimensions. You can identify the number of errors by thestatus_codelabel. You can use it to calculate the percentage of erroneous calls by dividing the metrics that contain astatus_codelabel equal toSTATUS_CODE_ERRORby the total number of metrics and multiplying it by 100.This is an example of the PromQL metrics exposed for thecalls_totalmetric:calls_total{operation=""/"", service_name=""digit"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
30255
calls_total{operation=""/"", service_name=""special"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_ERROR""}
84
calls_total{operation=""/"", service_name=""upper"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
15110
calls_total{operation=""GET /"", service_name=""lower"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
15085latency: This metric is made up of multiple underlying Prometheus metrics that combined represent a histogram. Because of the labels attached to these metrics, you can create histograms for latency on a per operation or service level.latency_count: thelatency_countcontains the total amount of data points in the buckets.latency_count{operation=""/"", service_name=""digit"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
32278
latency_count{operation=""/"", service_name=""special"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_ERROR""}
88
latency_count{operation=""/"", service_name=""upper"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_ERROR""}
87
latency_count{operation=""/"", service_name=""upper"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
16109latency_sum:thelatency_sumcontains the sum of all the data point values in the buckets.latency_sum{operation=""/"", service_name=""digit"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
8789319.72
latency_sum{operation=""/"", service_name=""special"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_ERROR""}
991.98
latency_sum{operation=""/"", service_name=""upper"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_ERROR""}
1087.28
latency_sum{operation=""/"", service_name=""upper"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
1060574.74latency_bucket:Thelatency_bucketcontains the number of data points where the latency is less than or equal to a predefined time. You can configure the granularity (or amount of buckets) by changing thelatency_histogram_bucketsarray in your OpenTelemetry Collector configuration.latency_bucket{le=""+Inf"", operation=""/"", service_name=""digit"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
32624
latency_bucket{le=""+Inf"", operation=""/"", service_name=""special"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_ERROR""}
90
latency_bucket{le=""+Inf"", operation=""/"", service_name=""special"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_UNSET""}
16150
latency_bucket{le=""+Inf"", operation=""/"", service_name=""upper"", span_kind=""SPAN_KIND_SERVER"", status_code=""STATUS_CODE_ERROR""}
88Learn more about the different types of Prometheus metrics in this blog post.How to Query the Prometheus MetricsYou can aggregate the Prometheus metrics mentioned above into RED metrics: RED stands for Rate, Error, and Duration. You can use these three metrics to identify slow and erroneous services within your distributed system.The Jaeger UI queries these metrics from Prometheus and visualizes them in three distinct service-level graphs. Latency (duration), Error rate (error), and Request rate (rate). On top of that, the Jaeger UI also presents you with these metrics per operation for the selected service.Because of its relative simplicity and lack of fine-grained controls, Jaeger’s SPM doesn’t eliminate the need for a more conventional metrics collection infrastructure and custom Grafana dashboards. But it should definitely be the first destination for anyone wanting to see what is going on in their distributed system at a glance on a per-service basis.Building on SPM: Creating Custom Instrumentation for Jaeger Distributed Tracing Using GrafanaSince these RED metrics are stored in a Prometheus-compatible storage backend, you can also query them outside the Jaeger UI. This is great if you combine these metrics with other custom instrumentation inside a Grafana dashboard.The following PromQL query gives us a simple overview of the request rate of all our services in a single Grafana panel:sum(rate(calls_total[1m])) by (service_name)Combining the following PromQL queries gives us a comprehensive view of the latency on a per percentile basis. By adding a variable in your Grafana dashboard, you can make it easier to switch between services.histogram_quantile(0.50, sum(rate(latency_bucket{service_name =~ ""generator""}[1m])) by (service_name, le))
histogram_quantile(0.90, sum(rate(latency_bucket{service_name =~ ""generator""}[1m])) by (service_name, le))
histogram_quantile(0.95, sum(rate(latency_bucket{service_name =~ ""generator""}[1m])) by (service_name, le))Setting Up an SPM-Compatible Jaeger Tracing DeploymentFor a deeper insight into our services, we will set up an SPM-compatible tracing deployment using a demo environment that runs onDocker Compose, and that is available in thePromscale repository.This deployment includes a demo microservices application that sends OpenTelemetry trace spans to the OpenTelemetry Collector, which includes the Spanmetrics Processor. The OpenTelemetry Collector sends spans to Jaeger, which stores them in Promscale, and sends the metrics generated by the Spanmetrics Processor to Promscale. We configured Jaeger to query and visualize traces and metrics in Promscale and deploy other components, but they are neither used nor required to demonstrate the Jaeger SPM capabilities.This is the architecture of the demo:You can get the environment up and running on your laptop very quickly. Just open a terminal and run the following commands:Once the setup is up and running, openhttp://localhost:16686in a browser to access the Jaeger UI. Navigate to theMonitortab to see the SPM user interface.1. Deploy TimescaleDB.services:
 timescaledb:
   image: timescale/timescaledb-ha:pg14-latest
   restart: on-failure
   ports:
     - 5432:5432/tcp
   volumes:
     - timescaledb-data:/var/lib/postgresql/data
   environment:
     POSTGRES_PASSWORD: password
     POSTGRES_USER: postgres
     POSTGRES_DB: tsdb
     POSTGRES_HOST_AUTH_METHOD: trust2.Deploy the Promscale Connector.Take note of the environment variable calledPROMSCALE_DB_URI, which points to the TimescaleDB service we configured in step one.promscale:
   image: timescale/promscale:latest
   restart: on-failure
   ports:
     - 9201:9201/tcp
     - 9202:9202/tcp
   depends_on:
     - timescaledb
   environment:
     PROMSCALE_DB_URI: postgres://postgres:password@timescaledb:5432/tsdb?sslmode=allow
     PROMSCALE_PKG: ""docker-quick-start""3.Deploy the OpenTelemetry Collector.The most important part of this snippet is the volume containing theotel-collector-config.yml. Because this is a crucial part of the deployment, we go into more detail in the Configuration section.collector:
   image: ""otel/opentelemetry-collector-contrib:0.63.1""
   restart: on-failure
   command: [ ""--config=/etc/otel-collector-config.yml"" ]
   depends_on:
     - promscale
   ports:
     - 14268:14268/tcp # jaeger http
     - 4317:4317/tcp
     - 4318:4318/tcp
   volumes:
     - ${PWD}/../otel-collector-config.yml:/etc/otel-collector-config.yml4.Deploy Jaeger.In the first two environment variables, we configure the tracing storage, which points to our Promscale service on port9202and uses thegrpc-pluginto do so. The third and fourth environment variables configure the Prometheus metrics storage, which also points to our Promscale service, but on port9201usinghttp.In this case, we use the all-in-one container, but it is possible to run thejaeger-collectorandjaeger-querycontainers separately, given you point them both to Promscale.jaeger-all-in-one:
   image: jaegertracing/all-in-one:1.39.0
   restart: on-failure
   environment:
     SPAN_STORAGE_TYPE: grpc-plugin
     GRPC_STORAGE_SERVER: promscale:9202
     METRICS_STORAGE_TYPE: prometheus
     PROMETHEUS_SERVER_URL: ""http://promscale:9201""
 
   depends_on:
   - timescaledb
   - promscale
   ports:
     - ""16686:16686""5. Deploy the microservices application.These services make up a distributed password generator. Thegeneratorservice is the entry point. Theloadservice continuously makes requests to thegeneratorservice to generate an artificial load. All the services are instrumented and export their traces to the OpenTelemetry service at port4317. You can find more information about these serviceshere.upper:
   image: timescale/promscale-demo-upper
   restart: on-failure
   depends_on:
     - collector
   ports:
     - 5054:5000/tcp
   environment:
 
     - OTEL_EXPORTER_OTLP_ENDPOINT=collector:4317
 
 lower:
   image: timescale/promscale-demo-lower
   restart: on-failure
   depends_on:
     - collector
   ports:
     - 5053:5000/tcp
   environment:
     - OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
 
 special:
   image: timescale/promscale-demo-special
   restart: on-failure
   depends_on:
     - collector
   ports:
     - 5052:5000/tcp
   environment:
     - OTEL_EXPORTER_OTLP_ENDPOINT=collector:4317
  digit:
   image: timescale/promscale-demo-digit
   restart: on-failure
   depends_on:
     - collector
   ports:
     - 5051:5000/tcp
   environment:
     - OTEL_EXPORTER_OTLP_ENDPOINT=collector:4317
 
 generator:
   image: timescale/promscale-demo-generator
   restart: on-failure
   depends_on:
     - upper
     - lower
     - special
     - digit
   ports:
     - 5050:5000/tcp
   environment:
     - OTEL_EXPORTER_OTLP_ENDPOINT=collector:4317
 
 load:
  image: timescale/promscale-demo-load
   restart: on-failure
   depends_on:
     - generator6. Finally, we create a volume for our TimescaleDB database.volumes:
 timescaledb-data:We will look closely at the OpenTelemetry Collector configuration yaml.1.ReceiversWe configure our OpenTelemetry protocol (OTLP) receivers for both thegrpcandhttpprotocols. In our case, the default configuration suffices, but if necessary, you can add endpoints, allow cross-origin resource sharing (CORS), and more per theopentelemetry-collector documentation.receivers:
 otlp:
   protocols:
     grpc:
     http:2.ExportersHere we configure what happens to our traces and metrics after they have been collected and processed:The Jaeger traces are sent to thejaeger-all-in-oneservice where we configured Jaeger to store them in Promscale.Usually, the OpenTelemetry collector exposes the Prometheus metrics for a Prometheus instance to scrape them. Still, in this case, we have configured that Prometheus metrics are to be sent to thepromscaleservice on port9201through theprometheusremotewriteconfiguration.exporters:
 logging:
 jaeger:
   endpoint: jaeger-all-in-one:14250
   tls:
     insecure: true
 prometheusremotewrite:
   endpoint: ""http://promscale:9201/write""
   tls:
     insecure: true3.ProcessorsProcessors run on data between being collected and exported.Thebatchprocessor collects traces and metrics and compresses the data reducing the number of outgoing requests.Thespanmetricsprocessor aggregates RED (Request, Error, Duration) metrics from collected traces. We configure this to send the aggregated data to theprometheusremotewriteexporter we configured earlier.processors:
 batch:
 spanmetrics:
   metrics_exporter: prometheusremotewrite4. ServiceLastly, we configure pipelines that define our data flow within the OpenTelemetry Collector:Thetracespipeline handles our traces. It collectsotlptraces from theotlpreceiver. It processes them in thebatchandspanmetricsprocessors.The metrics generated from thespanmetricsprocessor are handled internally through theprometheusremotewriteexporter and do not need to be defined again.Our traces are exported via thejaegerexporter after being batched in thebatchprocessor. The conversion of ourotlptojaegertraces happens automatically based on our configuration.service:
 pipelines:
   traces:
     receivers: [otlp]
     processors: [batch, spanmetrics]
     exporters: [jaeger]
   metrics:
     receivers: [otlp]
     processors: [batch]
     exporters: [logging, prometheusremotewrite]The Verdict: How Jaeger's SPM Changes Distributed TracingIn conclusion, Jaeger’s SPM feature is a welcome addition to the Jaeger UI. It provides easy access to RED metrics aggregated from traces you are already collecting.It is a great place to start troubleshooting your distributed system as it gives you a high-level overview of your service-level operations and the option to jump directly into Jaeger's Tracing tab on the operation of your choice for more detailed analysis. If you are already using the OpenTelemetry Collector to collect and process metrics and traces, enabling this feature is a no-brainer that takes no less than 10 minutes to configure.A way to further ease this transition is to store your metrics and traces in the same unified location. Operating and managing just a single storage backend reduces architectural complexity and operational overhead. Additionally, you benefit from correlating metrics and traces much more efficiently as they are stored in one place.This is why we recommend using PostgreSQL for Jaeger with Promscale.Promscale is a unified metric and trace storage for Prometheus, Jaeger, and OpenTelemetry built on PostgreSQL and TimescaleDB. With Promscale, you get a centralized and reliable long-term storage for your metrics and traces that offers the following:Full Jaeger support: passes all the Jaeger storage certification tests, has native support for OpenTelemetry and can be used as the metric storage backend for Jaeger’s Service Performance Monitoring (SPM) feature.First-class Prometheus support: fully PromQL-compliant and support for PromQL alerts and recording rules, exemplars, Prometheus high availability, and multi-tenancy.Flexible storage:configurable downsampling and retention policies, including per-metric retention, data backfilling and deletion, and full support for both PromQL and SQL.Rock-solid foundation: built on the maturity of PostgreSQL and TimescaleDB with millions of instances worldwide. A trusted system offering scalability, high availability, replication, and data integrity.Want to try it out? The easiest way to get started is tosign up for Timescale (create a free 30-day account, no credit card required).Self-hostingis also available for free.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/what-is-distributed-tracing-and-how-jaeger-tracing-is-solving-its-challenges/
2022-12-14T14:37:00.000Z,How to Turn Timescale Into an Observability Backend With Promscale,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.(This blog post was originally published in February 2022 and updated in December of the same year to include a different deployment option: installing Promscale using a Helm Chart.)The adoption of modern cloud-native distributed architectures has grown dramatically over the last few years due to the great advantages they present if compared with traditional, monolithic architectures—like flexibility, resilience against failure, or scalability.However, the price we pay is increased complexity. Operating cloud-native microservices environments is challenging: the dynamic nature of those systems makes it difficult to predict failure patterns, leading to the emergence of observability as a practice. The promise of observability is to help engineering teams quickly identify and fix those unpredicted failures in production, ideally before they impact users, giving engineering teams the ability to deliver new features frequently and confidently.A requirement to get the benefits of observability is access to comprehensive telemetry about our systems, which requires those systems to be instrumented. Luckily, we have great open-source options that make instrumentation easier—particularlyPrometheus exportersandOpenTelemetry instrumentation.Once a system is instrumented, we need a way to efficiently store and analyze the telemetry it generates. And since modern systems typically have many more components than traditional ones and we have to collect telemetry about each of these systems to ensure we can effectively identify problems in production, we end up managing large amounts of data.For this reason, the data layer is usually the most complex component of an observability stack, especially at scale. Often, storing observability data gets too complex or too expensive. It can also get complicated to extract value from it, as analyzing this data may not be a trivial task. If that’s your experience, you won’t be getting the full benefits of observability.In this blog post, we explain how you can use Timescale and Promscale to store and analyze telemetry data from your systems instrumented with Prometheus and OpenTelemetry.Integrating Timescale and Promscale: Basic ConceptsThe architecture of the observability backend based on Promscale and Timescale is quite simple, having only two components:The Promscale Connector.This stateless service provides the ingest interfaces for observability data, processing that data appropriately to store it in a SQL-based database. It also provides an interface to query the data with PromQL. The Promscale Connector ingests Prometheus metrics, metadata, and OpenMetrics exemplars using the Prometheusremote_writeinterface. It also ingests OpenTelemetry traces using the OpenTelemetry protocol (OTLP) and Jaeger traces using Jaeger gRPC endpoint. You can also ingest traces and metrics in other formats using the OpenTelemetry Collector. For example, you can use the OpenTelemetry Collector with the desired receivers and export the data to Promscale using Prometheus, Jaeger, and OpenTelemetry exporters.A Timescale service(i.e.,a cloud TimescaleDB database).This is where we will store our observability data, which will already have the appropriate schema thanks to the processing done by the Promscale Connector.Diagram representing the different components of the observability stack, where OpenTelemetry, Prometheus, Promscale, Jaeger, and Grafana are running in a Kubernetes cluster, and the observability data is stored in TimescaleCreating a Timescale ServiceBefore diving into the Promscale Connector, let’s first create a Timescale service (i.e., a TimescaleDB instance) to store our observability data:If you are new to Timescale,create an account(free for 30 days, no credit card required) and log in.Once you’re on the Services page, click on “Create service” in the top right, and select “Advanced options.”A configuration screen will appear, in which you will be able to select the compute and storage of your new service. To store your observability data, we recommend you allocate a minimum of 4 CPUs, 16 GB of Memory, and 300 GB of disk (equivalent to 5 TB of uncompressed data) as a starting point, this supports 50k samples per second. Once your data ingestion and query rate increase, you can scale up this setup as you need it.Hereis the resource recommendation guide for Promscale.Once you’re done, click on “Create service.”Wait for the service creation to complete, and copy the service URL highlighted with the red rectangle in the screenshot below. You will need it later!In Timescale, your service URL will be displayed right after creating your serviceNow that your Timescale service is ready, it is time to deploy the Promscale Connector on your Kubernetes cluster. We will discuss two different deployment options:Installing Promscale using Helm chart. This method requires that you are already running Prometheus, OpenTelemetry, or Jaeger in your Kubernetes cluster. With the Promscale Helm chart, you only need a single command to get started.Installing Promscale manually through a Kubernetes manifest. You can use this option if you are already running Prometheus or OpenTelemetry in your Kubernetes cluster.The above-listed deployment options will require that you manually configure the existing Prometheus, OpenTelemetry, Grafana, and Jaeger tools to connect to Promscale.Installing Promscale Using a Helm ChartIf you are already running Prometheus and/or OpenTelemetry in your Kubernetes cluster, you may prefer to use the Kubernetes manifest below, which will only install the Promscacle Connector. (The Promscale Connector is a single stateless service, so all you have to deploy is the connector and the corresponding Kubernetes service.)Promscale is a Helm chart that makes it simple to install the Promscale Connector.To install the Promscale Helm chart, follow these steps:Add Timescale helm repository.helm repo add timescale https://charts.timescale.com/2.  Update the helm repository.helm repo update3.   Now install the Promscale Helm chart. Through the command below, we are also configuring thetimescale-uri:We will connect the Promscale Connector to Timescale through the service URL you obtained when you created it.helm install promscale timescale/promscale --set connection.uri=<DB-URI>Note: Remember to replace <DB-URI> with the service URL from your Timescale service.4.   Once the installation is complete, you are good to go—Promscale is ready! Jump straight to the Prometheus, Grafana, and Jaeger sections to see how you can access and configure these tools.Install Promscale Using a Kubernetes ManifestIf you are already running Prometheus and/or OpenTelemetry in your Kubernetes cluster, you may prefer to use the Kubernetes manifest below, which will only install the Promscacle Connector. (The Promscale Connector is a single stateless service, so all you have to deploy is the Connector and the corresponding Kubernetes service.)Note: Remember to replace <DB-URI> with the service URL from your Timescale service.---
# Source: promscale/templates/service-account.yaml
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: false
metadata:
  name: promscale
  namespace: default
  labels:
    app: promscale
    app.kubernetes.io/name: ""promscale-connector""
    app.kubernetes.io/version: 0.16.0
    app.kubernetes.io/part-of: ""promscale-connector""
    app.kubernetes.io/component: ""connector""
---
# Source: promscale/templates/secret-connection.yaml
apiVersion: v1
kind: Secret
metadata:
name: promscale
  namespace: default
  labels:
    app: promscale
    app.kubernetes.io/name: ""promscale-connector""
    app.kubernetes.io/version: 0.16.0
    app.kubernetes.io/part-of: ""promscale-connector""
    app.kubernetes.io/component: ""connector""
stringData:
  PROMSCALE_DB_URI: ""<DB-URI>""
---
# Source: promscale/templates/svc-promscale.yaml
apiVersion: v1
kind: Service
metadata:
  name: promscale-connector
  namespace: default
  labels:
    app: promscale
    app.kubernetes.io/name: ""promscale-connector""
    app.kubernetes.io/version: 0.16.0
    app.kubernetes.io/part-of: ""promscale-connector""
    app.kubernetes.io/component: ""connector""
spec:
  selector:
    app: promscale
  type: ClusterIP
  ports:
  - name: metrics-port
    port: 9201
    protocol: TCP
  - name: traces-port
    port: 9202
    protocol: TCP   
---
# Source: promscale/templates/deployment-promscale.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: promscale
  namespace: default
  labels:
    app: promscale
     app.kubernetes.io/name: ""promscale-connector""
    app.kubernetes.io/version: 0.16.0
    app.kubernetes.io/part-of: ""promscale-connector""
    app.kubernetes.io/component: ""connector""
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: promscale
  template:
    metadata:
      labels:
        app: promscale
        app.kubernetes.io/name: ""promscale-connector""
        app.kubernetes.io/version: 0.16.0
        app.kubernetes.io/part-of: ""promscale-connector""
        app.kubernetes.io/component: ""connector""
      annotations: 
        prometheus.io/path: /metrics
        prometheus.io/port: ""9201""
        prometheus.io/scrape: ""true""
    spec:
      containers:
        - image: timescale/promscale:0.16.0
          imagePullPolicy: IfNotPresent
          name: promscale-connector
          envFrom:
          - secretRef:
              name: promscale
          ports:
            - containerPort: 9201
              name: metrics-port
            - containerPort: 9202
              name: traces-port
      serviceAccountName: promscaleTo deploy the Kubernetes manifest above, run:kubectl apply -f <above-file.yaml>And check if the Promscale Connector is up and running:kubectl get pods,services --selector=app=promscaleConfiguring PrometheusPrometheusis a popular open-source monitoring and alerting system used to easily and cost-effectively monitor modern infrastructure and applications. However, Prometheus is not focused on advanced analytics. By itself, Prometheus doesn’t provide durable, highly available long-term storage.One of the top advantages of Promscale is thatit integrates seamlessly with Prometheus for the long-term storage of metrics. Apart from its 100 % PromQL compliance,multi-tenancy, and OpenMetrics exemplars support, Promscale allows you to use SQL to analyze your Prometheus metrics: this enables more sophisticated analysis than what you’d usually do in PromQL, making it easy to correlate your metrics with other relational tables for an in-depth understanding of your systems.Manually configuring Promscale as remote storage for Prometheus only requires a quick change in the Prometheus configuration. To do so, open the Prometheus configuration file and add or edit these lines:remote_write:
  - url: ""http://promscale-connector.default.svc.cluster.local:9201/write""
remote_read:
  - url: ""http://promscale-connector.default.svc.cluster.local:9201/read""
    read_recent: trueCheck outour documentationfor more information on how to configure the Prometheus remote-write settings to maximize Promscale metric ingest performance!Configuring OpenTelemetryOur vision for Promscale is to create a unified interface where developers can analyzeall their data. How? By enabling developers to store all observability data (metrics, logs, traces, metadata, and other data types) in a single, mature, open-source, and scalable store based on PostgreSQL.Getting closer to that vision,Promscale includes beta support for OpenTelemetry traces. Promscale exposes an ingest endpoint that is OTLP-compliant, enabling you to directly ingest OpenTelemetry data, while other tracing formats (like Jaeger, Zipkin, or OpenCensus) can also be sent to Promscale through the OpenTelemetry Collector.If you want to learn more about traces in Promscale, watch Ramon Guiu (VP of Observability at Timescale) and Ryan Booz (former senior developer advocate) chat about traces, OpenTelemetry, and our vision for Promscale in the following stream:To manually configure the OpenTelemetry collector, we will add Promscale as the OTLP backend store for ingesting the traces that are emitted from the collector, establishing Promscale as the OTLP exporter endpoint:exporters:
  otlp:
    endpoint: ""promscale-connector.default.svc.cluster.local:9202""
    insecure: trueNote: In the above OTLP exporter configuration, we are disabling TLS setting insecure to true for the demo purpose. You can enable TLS by configuring certificates at both OpenTelemetry-collector and Promscale. In TLS authentication Promscale acts as the server.To export data to an observability backend in production, we recommend that you always use the OpenTelemetry Collector. However, for non-production setups, you can send data from the OpenTelemetry instrumentation libraries and SDKs directly to Promscale using OTLP. In this case, the specifics of the configuration depend on each SDK and library—see the corresponding GitHub repository or theOpenTelemetry documentationfor more information.Installing Jaeger QuerySince our recent contribution to Jaeger, it now supports querying traces from a compliant remote gRPC backend store and the local plugin mechanism. Now, you can directly use upstream Jaeger 1.30 and above to visualize traces from Promscale without the need to deploy our Jaeger storage plugin.A huge thank you to the Jaeger team for accepting our PR!In Jaeger, you can use the filters in the left menu to retrieve individual traces, visualizing the sequence of spans that make up an individual trace. That is useful to troubleshoot individual requests.Visualizing Promscale traces in the Jaeger UITo deploy the Jaeger query and Jaeger query service, use the manifest below:---
# Jaeger Promscale deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
  namespace: default
  labels:
    app: jaeger
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jaeger
  template:
    metadata:
      labels:
        app: jaeger
    spec:
      containers:
        - image: jaegertracing/jaeger-query:1.30
          imagePullPolicy: IfNotPresent
          name: jaeger
          args:
          - --grpc-storage.server=promscale-connector.default.svc.cluster.local:9202
          - --grpc-storage.tls.enabled=false
          - --grpc-storage.connection-timeout=1h
          ports:
            - containerPort: 16686
              name: jaeger-query
          env:
            - name: SPAN_STORAGE_TYPE
              value: grpc-plugin
---
# Jaeger Promscale service
apiVersion: v1
kind: Service
metadata:
  name: jaeger
  namespace: default
  labels:
    app: jaeger
spec:
  selector:
    app: jaeger
  type: ClusterIP
  ports:
  - name: jaeger
    port: 16686
    targetPort: 16686
    protocol: TCPTo deploy the manifest, run:kubectl apply -f <above-manifest.yaml>Now, you can access Jaeger:# Port-forward Jaeger service
kubectl port-forward svc/jaeger 16686:16686

# Open localhost:16686 in your browserConfiguring Data Sources in GrafanaAs shown in the following screenshot, to manually configure Grafana, we’ll be adding three data sources: Prometheus, Jaeger, and PostgreSQL.This screenshot shows the three data sources we’ll be configuring in Prometheus: Promscale-PromQL (the Prometheus data source for querying Promscale using PromQL), Promscale-Tracing ( the Jaeger data source for querying traces from Promscale), and Promscale-SQL (the PostgreSQL data source for querying TimescaleDB)Configuring the Prometheus data source in GrafanaIn Grafana navigate toConfiguration→Data Sources→Add data source→Prometheus.Configure the data source settings:In theNamefield, type Promscale-Metrics.In theURLfield, typehttp://promscale-connector.default.svc.cluster.local:9201, using the Kubernetes service name of the Promscale Connector instance. The 9201 port exposes the Prometheus metrics endpoints.Use the default values for all other settings.Note: If you are running Grafana outside the Kubernetes cluster where Promscale is running, do not forget to change the Promscale URL to an externally accessible endpoint from Grafana.Once you have configured Promscale as a Prometheus data source in Grafana, you can create panels populated with data using PromQL, as in the following screenshot:Example of a Grafana dashboard built querying Promscale with PromQLConfiguring the Jaeger data source in GrafanaIn Grafana, navigate toConfiguration→Data Sources→Add data source→Jaeger.Configure the data source settings:In theNamefield, typePromscale-Traces.In theURLfield, typehttp://jaeger.default.svc.cluster.local:16686, using the Kubernetes service endpoint of the Jaeger Query instance.Use the default values for all other settings.Note:The Jaeger data source in Grafana uses the Jaeger Query endpoint as the source, which in return queries the Promscale Connector to visualize the traces: Jaeger data source in Grafana -> Jaeger Query -> Promscale.You can now filter and view traces stored in Promscale using Grafana. To visualize your traces, go to the “Explore” section of Grafana. You will be taken to the traces filtering panel.Exploring traces from Promscale in GrafanaConfiguring the PostgreSQL data source in GrafanaIn Grafana, navigate toConfiguration→Data Sources→Add data source→PostgreSQL.Configure the data source settings:In theNamefield, typePromscale-SQL.In theHostfield, type<host>:<port>, where host and port need to be obtained from the service URL you copied when you created the Timescale service. The format of that URL is postgresql://[user[:password]@][host][:port][/dbname][?param1=value1&...]In theDatabasefield, type the dbname from the service URL.In the User and Password fields, type the user and password from the service URL.Change theTLS/SSL Modeto require as the service URL by default contains the TLS mode as required.Change theTLS/SSL Methodto File system path.Use the default values for all other settings.In the PostgreSQL details section, enable the TimescaleDB option.You can now create panels that use Promscale as a PostgreSQL data source, using SQL queries to feed the charts:Visualizing observability data in Grafana by querying from TimescaleDB using SQLUsing SQL to Query Metrics and TracesA powerful advantage of transforming Prometheus metrics and OpenTelemetry traces into a relational model is that developers can use the power of SQL to analyze their metrics and traces.This is especially relevant in the case of traces. Even if traces are essential to understanding the behavior of modern architectures, tracing has seen significantly less adoption than metrics monitoring—at least, until now. Behind the low adoption were difficulties associated with instrumentation, a situation that has improved considerably thanks to OpenTelemetry.However, traces were also problematic in another way: even after all the instrumentation work, developers realized there is no clear way to analyze tracing data through open-source tools. For example, tools like Jaeger offer a fantastic UI for basic filtering and visualizing individual traces, but they don’t allow analyzing data by running arbitrary queries or in aggregate to identify behavior patterns.In other words, many developers felt that adopting tracing was not worth the effort, considering the value of the information they could get from them. Promscale aims to solve this problem by giving developers a familiar interface for exploring their observability data and where they can use JOINs, subqueries, and all the advantages of the SQL language.In this section, we’ll show you a few examples of queries you could use to get direct value from your OpenTelemetry tracesandyour Prometheus metrics. Promscale is 100 % PromQL-compliant, but the ability to query Prometheus with SQL helps you answer questions that are impossible to answer with PromQL.Querying Prometheus metrics with SQLExample 1: Visualize the metricgo_gc_duration_secondsin GrafanaTo visualize such metric in Grafana, we would build a panel using the following query:SELECT
  jsonb(v.labels)::text as ""metric"",
  time AS ""time"",
  value as ""value""
FROM ""go_gc_duration_seconds"" v
WHERE
  $__timeFilter(""time"")
ORDER BY 2, 1The result would look like this:Grafana panel built with a SQL query graphing Prometheusgo_gc_duration_secondsExample 2: Calculate the 99th percentile over both time and series(pod_id)for the metricgo_gc_duration_secondsThis metric measureshow long garbage collection takes on Go applications. This is the query:SELECT 
   val(pod_id) as pod, 
   percentile_cont(0.99) within group(order by value) p99 
FROM 
   go_gc_duration_seconds 
WHERE 
   value != 'NaN' AND val(quantile_id) = '1' AND pod_id > 0 
GROUP BY 
   pod_id 
ORDER BY 
   p99 desc;And this is the result:A Grafana panel showing the p99 latency of garbage collection for all pods running Go applicationsWant more examples of how to query metrics with SQL?Check out our docs. ✨Querying OpenTelemetry traces with SQLExample 1: Show the dependencies of each service, the number of times the dependency services have been called, and the time taken for each requestAs we said before, querying traces can tell you a lot about your microservices. For example, look at the following query:SELECT
    client_span.service_name AS client_service,
    server_span.service_name AS server_service,
    server_span.span_name AS server_operation,
    count(*) AS number_of_requests,
    ROUND(sum(server_span.duration_ms)::numeric) AS total_exec_time
FROM
    span AS server_span
    JOIN span AS client_span
    ON server_span.parent_span_id = client_span.span_id
WHERE
    client_span.start_time > NOW() - INTERVAL '30 minutes' AND
    server_span.start_time > NOW() - INTERVAL '30 minutes' AND
    client_span.service_name != server_span.service_name
GROUP BY
    client_span.service_name,
    server_span.service_name,
    server_span.span_name
ORDER BY
    server_service,
    server_operation,
    number_of_requests DESC;Now, you have the dependencies of each service, the number of requests, and the total execution time organized in a table:Extracting service dependencies, number of requests, and total execution time for each API in a service.Example 2: List the top 100 slowest tracesA simple query like this would allow you to quickly identify requests that are taking longer than normal, making it easier to fix potential problems.SELECT
  start_time,
  replace(trace_id::text, '-', '') as trace_id,
  service_name,
  span_name as operation,
  duration_ms,
  jsonb(resource_tags) as resource_tags,
  jsonb(span_tags) as span_tags
FROM span
WHERE
  $__timeFilter(start_time) AND
  parent_span_id = 0
ORDER BY duration_ms DESC
LIMIT 100You can also do this with Jaeger. However, the sorting is done after retrieving the list of traces, which is limited to a maximum of 1,500. Therefore, if you have more traces, you could miss some of the slowest ones in the results.The result would look like the table below:List the slowest traces, including start time, trace id, service name, operation, duration, and tags. You could just click on the trace_id hyperlink for the traces listed in the table to inspect them furtherExample 3: Plot the p99 response time for all the services from tracesThe following query uses TimescaleDB’sapprox_percentileto calculate the 99th percentile of trace duration for each service by looking only at root spans(parent_span_id = 0)reported by each service. This is essentially the p99 response time by service as experienced by clients outside the system:SELECT
    time_bucket('1 minute', start_time) as time,
    service_name,
    ROUND(approx_percentile(0.99, percentile_agg(duration_ms))::numeric, 3) as p99
FROM span
WHERE
     $__timeFilter(start_time) AND
     parent_span_id = 0
GROUP BY time, service_name;To plot the result in Grafana, you would build a Grafana panel with the above query, configuring the data source as Promscale-SQL and selecting the format as “time series”.It would look like this:Graphing the p99 response time for all services in Grafana, querying tracing data using SQLGetting StartedIf you are new to Timescale and Promscale, follow these steps:You can install Promscalehere(it’s 100 % free) or get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.Deploy a full open-source observability stack into your Kubernetes cluster using tobs, our CLI. Apart from Promscale, you will also get tools like Prometheus, Grafana, OpenTelemetry, and Jaeger—fully configured and ready to go. 🔥Check out our documentationfor more information on the architecture of Promscale, Prometheus, and OpenTelemetry. You will also find tips for querying and visualizing data in Promscale.Check out thePromscale GitHub repo. Promscale is open-source and completely free to use. (GitHub ⭐️  and contributions are welcome and appreciated! 🙏)Explore our Youtube Channelfor tutorials on Timescale, TimescaleDB features like compression and continuous aggregates, Promscale, and much more!And whether you’re new to Timescale or an existing community member, we’d love to hear from you!Join us in our Community Slack: this is a great place to ask any questions on Timescale or Promscale, get advice, share feedback, or simply connect with the Timescale engineers. We are 8 K+ and counting!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-turn-timescale-cloud-into-an-observability-backend-with-promscale/
2022-05-25T11:14:00.000Z,OpenTelemetry: Where the SQL Is Better Than the Original,"This blog post was originally published at TFiR on May 2, 2022.⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.OpenTelemetry is a familiar term to those who work in the cloud-native landscape by now. Two years after the first beta was released it still maintains an incredibly active and large community, only coming second to Kubernetes when compared to other Cloud Native Computing Foundation (CNCF) projects.For those who aren’t so familiar, OpenTelemetry was born out of the need to provide a unified front for instrumenting code and collecting observability data—a framework that can be used to handle metrics, logs, and traces in a consistent manner, while still retaining enough flexibility to model and interact with other popular approaches (such as Prometheus and StatsD).This article explores how OpenTelemetry differs from previous observability tools, and how that point of difference opens up the potential for bringing back an old friend as the query language across all telemetry data.Observability—Then and NowAt a high level, the primary difference between OpenTelemetry and the previous generation of open-source observability tooling is one of scope. OpenTelemetry doesn’t focus on one particular signal type, and it doesn’t offer any storage or query capabilities. Instead, it spans the entire area that an application needing instrumentation cares about—the creation and transmission of signals. The benefit of this change in approach is that OpenTelemetry can offer developers a complete experience: one API and one SDK per language, which offers common concepts across metrics, logs, and traces. When developers need to instrument an app, they only need to use OpenTelemetry.On top of that promise, OpenTelemetry can take streams of signals and transform them, enrich them, aggregate them or route them, interfacing with any backend which implements the OpenTelemetry specification. This opens up a host of new deployment possibilities—a pluggable storage provider per signal (Prometheus, Jaeger, and Loki, maybe), a unified storage provider for all of them, two subsets of metrics to two different backends, or everything being sent out of a Kubernetes cluster to an external endpoint.Personally, the appeal of OpenTelemetry is very real to me—gathering telemetry data from a Kubernetes cluster using a single interface feels much more natural than maintaining multiple signal flows and potentially operators and custom resource definitions (CRDs). When I think back to the pain points of getting signals out of applications and into dashboards, one of my main issues was consistently around the fractured landscape of creating, discovering, and consuming telemetry data.OpenTelemetry and the Query Babel TowerWhen discussing OpenTelemetry, the question of querying signals soon comes up. It’s amazing we now have the ability to provide applications with a single interface for instrumentation, but what about when the time comes to use that information?If we store our data in multiple silos with separate query languages, all the value we gained from shared context, linking, and common attributes is lost. Because these languages have been developed (and are still being developed) for a single signal, they reinforce the silo approach. PromQL can query metrics, but it can’t reach out to logging or tracing data. It becomes clear that a solution to this problem is needed to allow the promise of OpenTelemetry to be realized from a consumption perspective.As it stands today, open-source solutions to this problem have mostly been offered via a user interface. For example, Grafana can allow you to click between traces and metrics that have been manually linked and correlate via time—but this soon starts to feel a bit limited.A New PromiseOpenTelemetry promises tagged attributes that could be used to join instrumentation and rich linkages between all signals. So what is the query equivalent of what OpenTelemetry promises? A unified language that can take inputs from systems that provide storage for OpenTelemetry data and allow rich joins between different signal types.This language would need to be multi-purpose, as it needs to be able to express common queries for metrics, traces, and logs. Ideally, it could also express one type of signal as another when required—the rate of entries showing up in a log stream which have a type of ERROR or a trace based on the time between metric increments.So, what would this language look like? It needs to be a well-structured query language that can support multiple different types of signal data; it needs to be able to express domain-specific functionality for each signal; it really needs to support complex and straightforward joins between data, and it needs to return data which the visualization layer can present. Other tools also need to support it, too. And hopefully, not just observability tools—integration with programming languages and business intelligence solutions would be perfect.Designing such a language is not easy. While the simplicity of PromQL is great for most metric use cases, adding on trace and log features would almost certainly make that experience worse. Having three languages that were similar (one for each signal) and could be linked together by time and attributes at query time is a possibility, but while PromQL is a de facto standard, it seems unlikely that LogQL (Grafana Loki’s PromQL-inspired query language for logs) will show up in other products. And, at the time of writing, traces don’t have a common language. Sure we could develop those three interfacing languages, but do we need to?Why SQL?Before working with observability data, I was in the Open Source database world. I think we can learn something from databases here by adopting the lingua franca of data analytics: SQL. Somehow, it has been pushed to the bottom of our programming languages kit but is coming back strong due to the increasing importance of data for decision-making.SQL is a truly a language that has stood the test of time:It’s a well-defined standard built for modeling relationships and then analyzing data.It allows easy joins between relations and is used in many, many data products.It is supported in all major programming languages, and if tooling supports external query languages, it’s a good bet it will support SQL as one of them.And finally, developersunderstandSQL. While it can be a bit more verbose than something like PromQL, it won’t need any language updates to support traces and metrics in addition to logs—it just needs a schema defined that models those relationships.Despite all this, SQL is a language choice that often raises eyebrows. It’s not typically a language favored by Cloud technologies and DevOps, and with the rise in the use of object-relational mapping libraries (ORMs), which abstract SQL away from developers, it’s often ignored. But, if you need to analyze different sets of data that have something in common—so they can be joined, correlated, and compared together—you use SQL.If before we dealt with metrics, logs, and traces in different (and usually intentionally simple) systems with no commonalities, today’s systems are becoming progressively more complex and require correlation. SQL is a perfect choice for this; in fact, this is what SQL was designed to do. It even lets us be sure that we can correlate data from outside of our Observability domain with our telemetry—all of a sudden we would have the ability to pull in reference data and enrich our signals past the labels we attach at creation time.At Timescale, we are convinced that a single, consistent query layer is the correct approach—and are investing in developing Promscale, a scalable backend to store all signal data which supports SQL as its native language. Whatever the solution is, we are looking forward to being able to query seamlessly across all our telemetry data, unlocking the full potential of OpenTelemetry.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/opentelemetry-where-sql-is-better-than-the-original/
2021-08-18T13:18:22.000Z,Simplified Prometheus monitoring for your entire organization with Promscale,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Computer infrastructure is rapidly moving to cloud-native architectures and Kubernetes. Observability is key to successfully operating systems in these highly dynamic environments andPrometheushas quickly become the de-facto standard for collecting and analyzing metrics. Engineers love its simple pull-based model with target auto-discovery for collecting multi-dimensional metrics, powerful query language (PromQL), and vibrant open-source community that has already builthundreds of integrationswith other tools.We love Prometheus for the above reasons and more – and it’s a key component of the observability stack we use to ensure the reliability ofTimescale, the modern, cloud-native relational database platform for time series built atop TimescaleDB.We are committed to making Prometheus the foundation of observability for large-scale systems everywhere.To this end, last October,we announced Promscale, an open-source, scalable, and robust storage system for Prometheus metric data. Built on top of the rock-solid foundation of TimescaleDB and PostgreSQL, Promscale inherits all of the capabilities of TimescaleDB and PostgreSQL, including full SQL support, advanced functionality for time-series analysis, and a robust ecosystem of tools, integrations, visualizations, and much more. Promscale is also100% PromQL-compliant, enabling developers to be immediately productive if they prefer to use Prometheus’ built-in query language.Since launching, we’ve found that developers love Promscale for two primary reasons:Promscale stores your data inTimescaleDB,a scalable, operationally mature, battle-tested time-series database built on top of PostgreSQL. As a result, users get built-in access to advanced database capabilities, like replication, data compression, hot backups, continuous aggregates, and many more.Promscale lets you perform in-depth data analysis using SQL (in addition to PromQL).Users combine PromQL’s benefits with SQL, or simply use SQL,one of the most popular and well-known languages in the world.As adoption of Prometheus in an organization grows, many teams set up individual Prometheus instances (let’s call themtenants) to monitor each Kubernetes cluster. Sometimes this happens organically, where one team is unaware of another team’s use of Prometheus, or it may happen by design, in order to obtain scale and robustness. No matter the reason, multi-tenant Prometheus introduces data silos, since Prometheus metrics across an organization now live in different data stores, and operational complexity, because all those data stores now need to be managed.To simplify operations and get a holistic view of their systems, organizations start to look for a centralized data store (i.e., one place for all of the metrics from all of the different tenants) – or build one themselves. Given its scalability, robustness, and advanced querying capabilities, Promscale is a natural fit for scenarios where users want to compare and contrast resource utilization, uptime, and performance across the entire organization.Today, we’re excited to announce multi-tenancy support in Promscale, available immediately🔥 .We’ve built Promscale as a centralized store for metrics, and now developers can enjoy the same operational maturity and query flexibility when storing metrics across their entire organization.With today’s release, Promscale now includes:Operational maturity: Centralizing data makes avoiding downtime even more critical. Building on top of PostgreSQL and TimescaleDB enables you to collect all of your important metrics with confidence.Advanced data analysis: Use PromQL and SQL (or both) to build queries that give you insight into the performance of your entire system.Faster and more robust cross-tenant queries:Effortlessly decorate your metrics with tenant-specific information, so that you can still easily query data by tenant, but also query data across tenants and across the organization.Flexible control and permissions: Customize which users can access metrics for specific tenants to ensure only authorized users see a tenant’s data. In addition to being useful for increased security, permissions make it easier for users to find the data they need vs. sorting throughallorganizational metrics.Read on for a primer on why we’ve built Promscale support for Prometheus multi-tenancy, the scenarios and challenges it solves for, the types of queries it frees you to make, as well as how to set up multi-tenancy for your team or organization.To get started right away:Install the latest version of Promscale, following the instructions in ourGitHub repository. As a reminder, Promscale is open-source and completely free to use.Join the TimescaleDB Slack community, where you’ll find 7K+ developers and Timescale engineers active in all channels. (The dedicated #promscale channel has 2.5K+ members, so it’s a great place to connect with like-minded community members, ask questions, get advice, and more).And, if these are the types of challenges you’d like to help solve, we are hiring (see all roles)!Introducing multi-tenancy support in PromscaleTo better understand the problem with multiple tenants described above, let’s look at a common scenario that occurs in medium-to-large organizations:The platform team of SomeCompany provides Kubernetes infrastructure to a number of development teams inside the company. Each team’s Kubernetes cluster is monitored with its own Prometheus instance. The platform team has agreed to internal SLAs with each of those development teams, and the platform team is responsible for ensuring each Kubernetes cluster is healthy and performs as expected.The platform team is also responsible for tracking and billing each team’s cost center for the fees associated with running their respective clusters. To do this, the platform team runs a service inside each Kubernetes cluster that generates billing information, stores the data as metrics in Prometheus, and then uses these billing metrics to measure and bill the teams on a per-cluster basis.To simplify operations and perform analysis across all clusters, the platform team wants to store all Prometheus metrics in a single (central) store. SomeCompany’s development teams have requested access to the telemetry and billing information for the clusters they own, but they don’t want to (and shouldn’t!) see data from other teams for various reasons. For example, individual development teams need to make sure the queries they run are always scoped totheirdata, so they don’t mistakenly query data from other tenants as they troubleshoot problems - which could lead to delays or faulty fixes - or analyze customer behavior - which could lead to making decisions that negatively impact the user experience for their product.This is a diagram of the architecture the platform team is envisioning:Example architecture diagram for a multi-tenant Prometheus use case.How can SomeCompany’s platform team solve for all of those use cases?The simplest option would be to use Prometheus and its default support for external labels. This would work to capture metrics collected in various Prometheus tenants, but it wouldn’t provide any security or access control guarantees without a custom solution bolted on top. (Even something as simple as ensuring that all incoming data includes a tenant’s label would require custom integration work!)Or, they could use Promscale with multi-tenancy enabled. 😎Promscale’s multi-tenancy support includes the ability to ingest metrics from different Prometheus tenants, label them with tenant information, query by tenant label, and restrict the tenants that a particular Promscale instance can ingest and query. Thus, SomeCompany’s platform team would be able to query data and billing metrics for specific tenantsandacross all tenants, while individual development teams’ queries would be “restricted” to the metrics from their tenant.How Promscale multi-tenancy support works and core featuresBefore we dive into specific steps, let’s look at a brief overview of how Promscale multi-tenant support works, basic architectural components, and what it allows developers to accomplish.To begin, each Prometheus instance needs to be configured to indicate which tenant it belongs to when setting upremote_write. This will ensure that the correct tenant is attributed when sending data to aPromscale Connector.Once configured and upon receiving metrics, the Promscale Connector ensures each metric is decorated with the respective__tenant__label so that the data can be differentiated by the tenant name. This information then makes it easy to use PromQL (or SQL) to write queries for specific tenants or across multiple tenants.To limit what data is available to a set of users, you set up an additional Promscale Connector and configure it to only allow query access to a specific tenant or group of tenants. Finally, you only allow those users to query data via that Promscale Connector. This is typically achieved by configuring a Prometheus data source for the Promscale Connector in Grafana, and then setting up the appropriate data source permissions.Core features and capabilities of multi-tenancy in Promscale include:Easy multi-tenancy configuration, using headers or Prometheus external labels, giving users flexibility in how they configure their systems.Cross-tenant queries in PromQL and SQL, streamlining how users query and analyze data across the entire organization.Support for combining data with and without tenant information in the same store, enabling users to evolve from a single-tenant to multi-tenant design, as well as support mixed deployments.Restricting the set of valid tenants a Promscale instance can ingest or query, enforcing access control.Configuring Prometheus to send tenant information to PromscaleFor multi-tenancy to work, you have to configure Prometheus to send tenant information to Promscale, so that Promscale can identify which tenant the data originated from.There are two ways to specify the tenant information, both done through the Prometheus configuration file:Pass a__tenant__label with all metrics (recommended)Use theTENANTHTTP header.See below for steps to configure Prometheus using either method.Pass a__tenant__label with all metrics (recommended)In Prometheus, you can leverage external labels for this. Prometheus will automatically add the__tenant__label to all metrics before they’re sent to Promscale.global:
 scrape_interval:    5s
 evaluation_interval: 30s
 external_labels:
   __tenant__: tenant-AUse theTENANTHTTP headerThe Prometheus configuration file allows you to set any number of HTTP headers to be sent with everyremote_writerequest to Promscale.remote_write:
-  url: http://localhost:9201/write
   headers:
     TENANT: team-1Once set up, the Promscale Connector will retrieve the value of theTENANTHTTP header. If that tenant is authorized in that Promscale Connector, Promscale will ingest and decorate all the metrics in the remote_write request by appending a__tenant__label. using the value of theTENANTheader.Enabling multi-tenancy in PromscaleYou can enable multi-tenancy in Promscale by setting thePROMSCALE_MULTI_TENANCY=trueenvironment variable when starting the Promscale Connector, or by passing the-multi-tenancyparameter.With this, Promscale will accept data from all tenants, both for write and read, and will add the corresponding__tenant__label to incoming metrics.If you want Promscale to allow ingest and query for data only from specific tenants, pass those tenant names separated by commas via the-multi-tenancy-valid-tenantsparameter or thePROMSCALE_MULTI_TENANCY_VALID_TENANTSenvironment variable.For example, if SomeCompany wants to allow Promscale to ingest and query dataonlyfor development teams 1 and 2, they’d set parameters like so:-multi-tenancy-valid-tenants=team-1,team-2With that setting, only data corresponding to team-1 or team-2 will be available, and Promscale will ignore and report all other data as unauthorized.Note:By default, the-multi-tenancy-valid-tenantshas the valueallow-all,  allowing all incoming tenants to be ingested and queried.When multi-tenancy is enabled, Promscale drops all data from a Prometheus instance that isn’t configured tosend tenant information. You instruct Promscale to ingest the data by passing the-multi-tenancy-allow-non-tenantsparameter or thePROMSCALE_MULTI_TENANCY_ALLOW_NON_TENANTS=trueenvironment variable when launching the Promscale Connector.Querying multi-tenant dataUsing PromQLAs explained in the previous section, multi-tenant data will include a special, defined__tenant__label.To filter by tenant name in PromQL, simply apply the tenant name(s) using the__tenant__label matcher for your query.For example, if SomeCompany wants to find the number of CPU-hours team 1 used and will be billed for, they’d run the following query:cpu_hours_total{__tenant__=”team-1”}This query returns all the time-series data with the metriccpu_hours_totalfrom team 1.From there, SomeCompany can run a cross-tenant query against two teams, team 1 and team 2, to calculate total CPU hours used across both teams:sum(cpu_hours_total{__tenant__=~”team-1|team-2”})This type of query becomes particularly useful if you have multiple teams in your organization and you want aggregate statistics using GROUP BY queries.For all queries, the-multi-tenancy-valid-tenantsflag will be respected and data will only be returned from allowed tenants whether or not the appropriate matchers are in the query.Checkour documentationfor more PromQL query examples.Using SQLAs we saw above, PromQL offers great ergonomics for querying observability data.With Promscale, you canalsouse SQL to do more sophisticated analysis and to correlate your Prometheus metrics with other relational data stored in the underlying PostgreSQL database.For example, suppose you wanted to use SQL to obtain the equivalent ofcpu_hours_total{__tenant__=”team-1”}in SQL (it returns the same data points but formatted differently):SELECT
   * 
FROM
   prom_metric.cpu_hours_total 
WHERE
    labels ? (‘__tenant__’ == ‘team-1’);You can also perform queries in SQL that are not possible in PromQL.In PromQL, all queries have to aggregate data within each time-series before doing other aggregations. But, when comparing results between tenants, this can skew results by weighing tenants unevenly by the number of series present.For example, when comparing the p95 latencies of HTTP requests by tenant, the grouping of requests by series is irrelevant since you want to compareallthe requests of one tenant with another. The SQL query for that is shown below (note that in our views, we expose identifiers for label values using the “_id” suffix, thus the__tenant__label’s id is the somewhat strange__tenant___id).SELECT
   val(__tenant___id) as tenant_name, --val() looks up the tenant name from the id
   percentile_cont(0.95) WITHIN GROUP (ORDER BY value) as p95,
FROM
   prom_metric.http_requests_total
GROUP BY __tenant___id      --grouping by the id is much more efficient than using the nameIt’s also possible to do even more sophisticated analysis, joining organizations’ relational data with Prometheus metrics! Joining time-series data with relational data has all manner of practical applications and is one of TimescaleDB and Promscale’s superpowers.For example, SomeCompany team 1 could see the total number of cpu hours they have received from their customers, by company size, in the last 30 days. (This example assumes the developmentcpu_hours_totalmetric includes acustomer_namelabel with the name of the customer, and that there is acustomerstable in PostgreSQL that includes the customer name and company size.)SELECT
   customers.company_size, sum(value)
FROM
   prom_metric.cpu_hours_total
JOIN customers ON customer.name = val(customer_name_id)
WHERE time > now() - INTERVAL ‘30 day’
GROUP BY
    customers.company_sizeJoining metric data with relational data can be used in a variety of use-cases: to analyze how infrastructure costs correlate with usage and/or profits; to compare performance with hardware specifications of the machines or with configuration parameters; to use inventory data to do predictive maintenance (on disk drives, for instance); or to find underutilized or overutilized resources.ConclusionIn summary, we built Promscale -- an operationally mature long-term data store for Prometheus metrics -- to solve common challenges that we, and many other organizations, face.With the addition of multi-tenancy, we’re making it easier to centralize Prometheus data across large, diverse organizations while tapping into PostgreSQL’s operational maturity and enabling developers to use PromQL and SQL to perform even more advanced queries.Centralizing Prometheus data unlocks a lot of potential for in-depth data analysis and organization-wide optimization, as well as reducing operational complexity.We’re excited to continue to identify and find new ways to help developers better query and manage their data.Get started with PromscaleIf you’re new to Promscale and want to get started with our new multi-tenancy functionality today:Install Promscale today via Helm Charts, Docker, and others.Follow the instructions in ourGitHub repository. As a reminder, Promscale is open-source and completely free to use. (GitHub ⭐️  welcome and appreciated! 🙏.)If you are looking for managed Prometheus storage, get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.See our docson how to enable multi-tenancy support on your Promscale instance today.Check out ourGetting Started with Promscale tutorialfor more on how Promscale works, installation instructions, sample PromQL and SQL queries, and more.WatchPromscale 101 YouTube playlistfor step-by-step demos and best practices.Whether you’re new to Promscale or an existing community member, we’d love to hear from you!Join TimescaleDB Slack, where you’ll find 7K+ developers and Timescale engineers active in all channels. (The dedicated #promscale channel has 2.5K+ members, so it’s a great place to connect with like-minded community members, ask questions, get advice, and more).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/simplified-prometheus-monitoring-for-your-entire-organization-with-promscale/
2021-02-09T11:32:23.000Z,"Introducing Prom-Migrator: 
A Universal, Open-Source Prometheus Data Migration Tool","⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.We've built a brand-new, 100% free, open-source tool that makes it easy to migrate your Prometheus metrics data to and from various long-term storage systems. Learn how it works and get started.Prometheus is a metric collection, alerting, and storage system. It is architected to store data locally in its own time-series database (TSDB) and/or to ship data to other long-term storage systems via its remote_write/remote_read APIs. Over time, many long-term storage systems have emerged, and, as a result, users can have data stored in a variety of systems. For example, we recently shippedPromscale, an analytical platform and long-term store for Prometheus, built on top of TimescaleDB. But switching between these storage systems has not been possible because, until now, there has been no universal tool to migrate data between them.Prom-migratorsolves this problem. It is a universal, open-source Prometheus data migration tool that is community-driven and free to use. Prom-migrator migrates data from one remote-storage system to another remote-storage system, leveraging Prometheus’ remote storage API. This means that this tool can be used to migrate data from any storage system that supports the Prometheus remote_read protocol. Similarly, it can migrate data to any storage system that supports the Prometheus remote_write protocol.In this post, we'll share why we built Prom-migrator, including the problems it solves, which systems it is compatible with, and how it works. We'll also take you through two short examples to show how you'd use Prom-migrator in two very different scenarios – and, in the process, hopefully inspire you to give it a try yourself.Why did we build Prom-migrator?Prometheus has many remote-storage systems and this has been growing over time.  However, there was no universal tool for data migration, leaving users with few good options if they wanted to switch between different remote-storage systems:They could throw away the old data they had in their previous system. This led to gaps in historical knowledge about a system.They could continue to run both the old and the new system and try to redirect relevant queries to either system. This led to operational and data management headaches.They could continue using the previous system and not switch. This is classic vendor lock-in.Prom-migrator featuresProm-migrator offers several new features:Data migration from and to any storage system.This tool is designed to work with any remote storage system, so that users can migrate data across any system in a wide range of scenarios.Informative outputs during runtime, allowing users to track progress.Prom-migrator keeps users informed about the migration progress, so that users can plan their time accordingly.Ability to resume migration(s) in case of any unintended shutdowns.Keep a record of migration progress and, in the case failure or interruptions, automatically resume the migration where you left off.Stateless working model. Easy deployment due to the fact that the migrator does not need to keep state. No need to worry about mounting volumes or attaching persistent storage.Compatibility between Prom-migrator and other databasesThe chart below describes the systems Prom-migrator can work with, both for reading and for writing data. We break out two cases for the migration destination endpoint: what we call “write,”  which is migrating data to an empty database and what we call “backfill,”  which is migrating data to a database that already has data newer than what is being migrated.As you can see, many remote storage systems support migrating into an empty database, but not backfill. This is because these systems expect data ingest in loose time order and cannot support out-of-order ingest. (Promscale is the exception here, as it does support backfill.)Key:Read - means data can be read from systemWrite - means data can be written to an empty databaseBackfill - means data can be written to a database that already contains data newer than what is being insertedStorage nameReadWriteBackfillPromscaleYesYesYesPrometheusYesYes(Work under progress,Experimental)No*Cortex (blocks storage)YesYesNo*Cortex (chunk storage)YesYesNoThanosYes(via G-research’s thanos-remote-read)YesNo*M3DBYesYesNoVictoria MetricsNoYesYesInfluxDB v1.8YesYesYes* you may be able touse promtoolto backfill data in these systemsIn general, all storage systems that support Prometheus’ remote_write are supported by Prom-migrator for writing to an empty database. Similarly, storage systems that support Prometheus’s remote_read endpoint are supported for reading data by the migration tool.How Prom-migrator worksLet’s dive deeper into how Prom-migrator works with your desired storage systems. Here’s a conceptual overview of the process:Prom-migrator migrates data from one storage to another. It iteratively pulls data from a remote storage system using the remote_read endpoint for a certain time-range.To start, Prom-migrator pulls data from remote storage systems, using the remote_read endpointThen, Prom-migrator pushes the data to another remote storage system using the remote_write endpoint. It then advances the time-range it is working on and repeats this process. This continues until it finishes the entire time-range specified by the user. The time-range of any individual read is adaptively auto-adjusted to bound the overall memory usage of the system.Once data is pulled, Prom-migrator then pushes data to the destination’s remote_write endpointThe system is able to auto-resume the migration if previously interrupted. This is done by adding a progress-metric to the data while it is writing.The sample of the progress-metric records the maximum time that was written. Thus, when a migration process resumes, it simply reads the progress-metric to find out what was last written and picks up where it left off.Prom-migrator tracks the maximum time written to the destination storage system in order to provide auto-resume capabilitiesFor detailed information about working, design and process, please refer to theProm-migrator design doc.Using Prom-migratorLet’s examine how you could use Prom-migrator to migrate your existing data in two scenarios:Migration from Prometheus’s time-series database (TSDB) to PromscaleMigration from one remote storage system to anotherMigration from Prometheus’ TSDB to PromscaleIn this case, we will show how to migrate data from Prometheus’ built-in time-series database to any remote storage system that supports a Prometheus-compliant remote_write endpoint. You may want to do this when you’re first adding a long-term storage system to your observability stack.In this scenario, your existing Prometheus data will show up in the system right away.Promscaleis the remote-storage solution provided by TimescaleDB, so we will illustrate this example by sending data to TimescaleDB using Promscale.Install and extract PrometheusYou can download the Prometheus binary from the GitHubreleases pageof the Prometheus repository. For setting up of Prometheus, you can visit the relatedPrometheus documentation.# Download the Prometheus binary
wget https://github.com/prometheus/prometheus/releases/download/v2.24.1/prometheus-2.24.1.linux-amd64.tar.gz

# Extract the contents of the binary
tar -zxvf prometheus-2.24.1.linux-amd64.tar.gzOnce you have extracted the downloaded tar file, you will notice theprometheus.ymlfile. This is our configuration file for running the Prometheus binary. For simply running Prometheus, the default configuration file (prometheus.yml) will not require any changes.Start Prometheus./prometheus --config.file=prometheus.ymlThis sets up the remote_read storage which we will be using as our source of data. Upon starting Prometheus, it will start scraping the targets mentioned in its configuration file and store the samples scraped into its local TSDB only (provided remote_write url is not mentioned).Setup PromscaleNow, let's set up Promscale. Promscale is a remote read/write storage platform that is offered by Timescale. It accepts Prometheus data via remote_write and stores it in TimescaleDB. Promscale enables Prometheus data to be queried and analyzed using PostgreSQL and PromQL natively (while remaining 100% PromQL compatible according to thePromQL compliance tests result from PromLabs).For setting up Promscale, refer to theinstallation sectionof Promscale. For this example, we will be using binaries from the GitHubreleases pageof Promscale.# Download the Promscale binary
wget -O promscale https://github.com/timescale/promscale/releases/download/0.1.4/promscale_0.1.4_Linux_x86_64

# Provide execution permissions
chmod +x promscaleAfter downloading the respective Promscale version, you need to start Promscale as mentioned in the PromscaleREADME. In this example, we want to run Promcale on bare metal.Start Promscale./promscale -db-name=<db-name> -db-password=<password> -db-ssl-mode=allowAfter setting up Prometheus and Promscale, we now have to supply three URLs as inputs to Prom-migrator:remote_read URLremote_write URLprogress-metric URLFor this example, these would correspond to:Source Read URL (Prometheus):http://localhost:9090/api/v1/readDestination Write URL (Promscale):http://localhost:9201/writeDestination Read URL for the Progress-metric (Promscale):http://localhost:9201/readMigrating data from Prometheus to Promscale using Prom-migrator.The progress metric is used internally by Prom-migrator to ensure that if our migration is interrupted, it can be automatically resumed from where it left off. Prom-migrator does this by reading the value of the progress-metric it wrote as part of the interrupted migration.Thus, in order to resume the migration, we need to tell the migrator where to read the value or the progress metric. Since the migrator writes the progress metric to the destination, the URL for fetching the progress metric will be theremote_read URLof Promscale, which serves as the input to progress-metric-url.Next, let’s set up the things necessary for performing the migration itself. We start by downloading the Prom-migrator’s binary from Promscale’s GitHubrelease page.Let's do the migrationAny migration requires the following:starttime from which the data migration is to be carried out.reader-url, the url of the storage system from which the data is to be read/fetched.writer-url, the url of the storage system where the data is to be pushed.Theendtime is an option field, as its default value is the current time. In this example, we want to migrate everything till now. So, we will leave the end time empty. Moreover, since  Promscale is the destination, we can use its remote_read url as theprogress-metric-url. For more information about various configurations of Prom-migrator, please refer to theProm-migrator docs.We execute the migration with the following command after ensuring that Prometheus and Promscale are running on the URLs mentioned above../prom-migrator -start=1608018121 -reader-url=http://<prometheus_host>:9090/api/v1/read -writer-url=http://<promscale_host>:9201/write -progress-metric-url=http://<promscale_host>:9201/readThe above command executes migration from the Prometheus instance running at :9090 to Promscale instance running at :9201, migrating data from 1608018121 (time in unix seconds) up to now and at the same time, maintaining the progress of the migration carried out, so that the process can be resumed in case of any interruption.Note:We did not specify theend(or maximum timestamp up to which migration should be carried out) since by default,endcorresponds to the current time, meaning all data fromstartupto now would be migrated.We’ve recorded the following video to walk you through the entire process:Vineeth takes you through how Prom-migrator works and how to get up and running in under 10 mins 🔥.Migration from one remote storage to anotherIn this second scenario,  we will discuss how to migrate data from one remote-storage system to another. For this example, we will transfer data from a Cortex instance to Promscale using Prom-migrator.Cortex supports a Prometheus-compliant remote_read endpoint. Hence, the read API in here will serve as the input to-reader-urlin Prom-migrator. For the-writer-url, we will use Promscale’s Prometheus-compliant remote_write endpoint. We also want to keep track of the progress so that we are protected from intentional crashes. Hence, we provide Promscale’s remote_read endpoint as the input to-progres-metric-urlso that prom-migrator can push the timestamp of the most recently migrated block.Let's break this down step by step.Getting startedIn this example we will assume that Cortex is already running. You can get information on how to set up Cortex from itsdocumentation.You will also need to download andinstall Promscaleaccording to the instructions found in the “Setup Promscale” section, above.Start both Promscale (“Start Promscale,"" above) and Cortex according to their respective instructions.Download Prom-migratorPromscale and Cortex are up and running. You can download Prom-migrator from Promscale’sreleasespage.Now we can begin the migration:./prom-migrator -reader-url=http://<cortex_host>:9009/api/prom/api/v1/read -writer-url=http://<promscale_host>:9201/write -start=1609920418 -progress-metric-url=http://<promscale_host>:9201/readThe above URL migrates the data from Cortex running at :9009 to Promscale running at :9201 from the timestamp 1609920418 to now. At the same time, the prom-migrator utilizes the progress-metric-url to maintain the progress of the migration carried out, so that the process can be resumed in case of any interruption.Note:We did not specify theend(or maximum timestamp up to which migration should be carried out) since by default,endcorresponds to the current migration, meaning all data from mint up to now needs to be migrated.With the above command, you should see blocks with progress bars being formed with their respective time-ranges and the progress percent as per the overall migration task.ConclusionPrometheusis an open-source systems monitoring and alerting toolkit that can be used to easily and cost-effectively monitor infrastructure and applications. Over the past few years, Prometheus has emerged as the monitoring solution for modern software systems. The key to Prometheus’ success is its pull-based architecture in combination with service discovery, which is able to seamlessly monitor modern, dynamic systems in which (micro-)services startup and shutdown frequently.Long-term storage of Prometheus metrics gives you greater insight into your systems. Unfortunately, Prometheus does not itself provide durable, highly-available long-term storage or advanced analytics, but relies on other projects to implement this functionality.Prom-migrator is a universal, open-source Prometheus data migration tool that's community-driven and free to use. With Prom-migrator, you can move your data from Prometheus to the long-term storage of your choice, or migrate between different long-term storage options.For more:Visit ourGitHub page(README) and try Prom-migrator today.File issues and PRswith your feedback and suggestions.Get involved and talk to us onPromscale’s Community Call(held on the 2nd Wednesday of each month), ask in the#promscale channel on Timescale DB Slack, or open a topic in thePromscale users Google Group.For more information about Prom-migrator’s design, please read ourpublic design doc.If you are looking for managed Prometheus storage, get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.Finally, we would love to collaborate with any remote-storage provider to improve Prom-migrator support for their tool.Please reach out to[email protected]or simply file issues and PRs in GitHub.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/introducing-prom-migrator-a-universal-open-source-prometheus-data-migration-tool/
2022-05-17T13:16:16.000Z,No-Fuss Alerting: Introducing Prometheus Alerts in Promscale,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.This is KubeCon Europe week, and at Timescale, we’re celebrating it the way we know best: with a new edition of #AlwaysBeLaunching! 🐯 🎊 During this week, we are releasing new features and content every day, all focused on Promscale and observability. Do not miss any of it by following our hashtag #PromscaleKubecon on Twitter—and if you’re at KubeCon, visit us at booth G3! We have awesome demos (and swag) to give you!Our launch week continues! Today, we’re excited to introduce you to a new capability in Promscale: the native support for alerts.Alerting is a crucial component of monitoring cloud-native architectures, as it allows developers to automatically identify anomalies and take action before they become problems.Prometheusis the most popular open-source tool for metrics monitoring and alerting. It gives users great flexibility and robust functionality for enabling alerting rules on their metrics using PromQL.Now, you can use the same standard todefine alerts directly within Promscale. You can load the same PromQLalerting rulesyou would use in Prometheus into Promscale, even using the same YAML configuration files. This will allow you to collect metrics using Prometheus Agent Mode, which needs fewer resources to run than a separate Prometheus instance. Plus, it also opens the door to streaming metrics from OpenTelemetry directly into Promscale and raising alerts on them.To learn how you can configure alerts in Promscale, keep reading. To install Promscale, clickhere. It’s 100 % free!✨ Promscale is the observability backend built on PostgreSQL and TimescaleDB. It has native support for Prometheus, 100 % PromQL compliance, native support for OpenTelemetry tracing, and seamless integration with Grafana and Jaeger. If you’re new to Promscale,check out our websiteanddocumentation.Configuring Alerting Rules in PromscaleIn Prometheus,alerting rulesallow you to define conditions for your metrics using PromQL, Prometheus’ query language. For example, you can specify a rule with the condition “alert me if a node has been using more than 90 % of its storage for five minutes,” or perhaps, “alert me if a queue is full and new items are being dropped.” Prometheus continuously evaluates these conditions, marking the alerting rule as “firing” if they are met.Once a condition is met and an alert is fired, Prometheus sends the alert to one (or multiple)Alertmanagerinstances. These are responsible for managing the alert lifecycle (grouping, suppression, deduplication, routing), including sending notifications via the medium of your choice (e.g., Slack, email, Pagerduty) when you need to notify a real user.Alerting Rules areconfigured in Promscalesimilarly to in Prometheus—using a YAML file that identifies the AlertManagers and links to YAML files containing alerting rules to load. In fact, the file formats are identical, so if you have a working Prometheus configuration that includes analerting:block and some alerting rules, you can point Promscale at the config, and things will work.So what does that config look like? A basic example of the config needed would be aprometheus.ymlwith the following content:# Alerting settings
alerting:
  alert_relabel_configs:
   - replacement: ""production""
     target_label: ""env""
     action: ""replace""
  alertmanagers:
    - static_configs:
      - targets:
        - alertmanager:9093

# Rules and alerts are read from the specified file(s)
rule_files:
  - alerts.ymlThis will configure Promscale to send all alerts to a single Alertmanager, adding anenv: productionlabel to each alert on the way out.Rules will be read from thealerts.yml, which contains a list of PromQL alerts to load and evaluate. An example of the file with a Watchdog alert that will always be firing would be:groups:
- name: alerts
  rules:

  - alert: Watchdog
    annotations:
      description: > 
        This is a Watchdog meant to ensure that the entire Alerting  
        pipeline is functional. It is always firing.
      summary: Alerting Watchdog
    expr: vector(1)Once we have the config files, Promscale can be started with either themetrics.rules.prometheus-configargument or thePROMSCALE_METRICS_RULES_PROMETHEUS_CONFIGenvironment variable pointing at theprometheus.ymlfile.Check Out an ExampleIf you’d like to give this a go, you can use the Docker Compose file inGitHub’s Promscale repositoryto see an alert working.If you execute the following commands from a shell:git clone https://github.com/timescale/promscale
cd promscale/docker-compose
docker-compose upYou should see the stack come up. After waiting for it to stabilize, you can browse to theAlertManager web interface(available at localhost:9093). After a short time, you will see an alert called Watchdog which is always firing (it’s normally used to make sure the alerting pipeline is working).Example of an alert configured in Promscale (""Watchdog"" alert) in the AlertManager UIAnd For Our Next Trick...We are really excited to provide this functionality to our Promscale users, allowing them to select where they produce alerts and giving them the option of not running a full Prometheus instance on edge. If you are not yet a user, you can install Promscale for freehereor get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.But while supporting alerting rules in PromQL is a significant step forward, that's only the start of Promscale’s alerting journey. In the future, we are planning to support sending alerts to Alertmanagers based on pure SQL queries. This will allow many new use cases outside of metric alerts—alerts on logs or alerts on trace content will be possible as we continue to grow our OpenTelemetry support.If you’ve got any ideas or questions on alerting, feel free to reach out! You can interact with the team building Promscale in ourCommunity Slack(make sure to join the #promscale channel). And for hands-on technical questions, you can also post in theTimescale Forum.See you at the next launch!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/introducing-prometheus-alerts-in-promscale/
2022-05-19T13:08:06.000Z,"How We Built Alert Rules, Runbooks, and Dashboards to Observe Our Observability Tool","⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.In this era of cloud-native systems, the focus on observability is a given: observability tools ensure your systems perform correctly and deliver a satisfactory experience to your end-users. In the absence of good observability, your end-users will be the first to notify you about a problem—a tiny dart piercing through every respectable developer’s heart.Needless to say, your users shouldn’t do your monitoring and alerting for you. You should aim to detect and fix problems before your users notice them, not after. When an issue arises, you need to find the root of the problem. And in a distributed architecture, the best way to do so is byinterrogating your telemetry data with random questions, getting results in real time. You needrealobservability.Observing the ObserverWe all know that observability tools are critical for any system that aims to deliver an always available and reliable service. But what happens if your observability tool stops working? If your system has an issue, you’d only notice it when a user of said system notifies you—meaning that we’d be back to square one, running blind again.As an observability practitioner, you’ll find many observability tools and tons of resources on configuring them tocollect,store,query,visualize, andalerton telemetry data from the systems you monitor.Butwho observes the observer?We must treat observability tools as highly available and reliable systems. They, too, have to be monitored to ensure correct behavior. But it is surprisingly hard to find information on how to observe your own observability tool effectively. For example, we’ve struggled to find information online about how to monitor Prometheus —if there is any, we have not been able to find it.This seems like a missing piece (more like a missing pillar) in our observability journey.In the Promscale team, we decided to prioritize this. As a result, we’ve built a set of alerting rules, runbooks, and dashboards that will help Promscale users track the performance of their own Promscale instance while providing them with guidance on how to fix common issues. In this blog post, we tell you how we did it.We relied on open-source components to build this set of tools, combined with our own experience assisting Promscale users. So even if you’re not a Promscale user, we hope this blog post can give you ideas on how to build your own “observing the observer” setup.Observing PromscaleTo understand our reasoning behind the alerts, runbooks, and dashboards we ended up creating, we’ll have to first explain in more detail how Promscale works.Promscale is a unified observability backend for metrics and traces built on PostgreSQL and TimescaleDB. It has a simple architecture with only two components: the Promscale Connector and the Promscale Database.The Promscale architectureThePromscale Connectoris a stateless service that adds a “plug and play” functionality to connect some of the most common observability tools and protocols to the Promscale Database. These are some of its functions:Ingestion ofPrometheus metricsandOpenTelemetry tracesPromQL query supportAPIs for seamlessintegration with the Jaeger and Grafanauser interfaces to visualize distributed tracesPromQLalertingandrecording rulesThe second component is thePromscale Database. The Promscale Database is PostgreSQL with observability superpowers, including:An optimized schema for observability data, includingmetricsandtracesAll thetime-series analytical functionsand the performance improvements provided by TimescaleDBDatabase management functions, aggregates to speed up PromQL queries, and SQL query experience enhancements for observability dataTools that speak SQL can connect directly to the Promscale Database, while other common open-source tools such as Prometheus, Jaeger, and Grafana can integrate with the Prometheus Connector.This simple architecture benefits easier troubleshooting. Promscale’s PostgreSQL foundation also helps—we’re talking about a very mature piece of software with a lot of documentation and knowledge around its configuration, tuning, and troubleshooting.Still, we knew that we could accelerate the production-readiness process by providing extra guidance to our users through an extensive set of alerts and runbooks created by the engineering team building the product.Common Performance BottlenecksFrom our conversations with users, we learned that when tracking Promscale’s performance, there are three processes that they should be paying particular attention to:Data ingestIn Promscale, metrics and traces follow different ingest paths. Let's cover them separately.MetricsWhen metrics are ingested, they are transformed into the Promscale metric schema. This schema stores the series’ metadata and data points in separate tables. Each data point includes the associated series’ identifier. Metric labels (both keys and values) are also stored in a different table: only the IDs that reference the values are stored in the series table.To avoid running queries to retrieve those IDs from the database when new data points are inserted for existing series, the Promscale Connector keeps a cache with all that information, including all labels for a series ID, as well as the corresponding IDs of all the keys and values that have already been seen. As new series are ingested, the cache is automatically updated.If the cache size is not enough to hold all the series information, the Promscale Connector will automatically increase the cache size up to a certain configurable limit. If that limit is hit and new series are ingested, Promscale will start evicting the oldest series from the cache. After eviction, the Promcale Connector will have to query the database to retrieve the series if it is ingested again.If the cache is too small to contain all the “active” series (i.e., series that are being ingested regularly), then the system will start a loop where a series is loaded into the cache, evicted, and re-loaded again at the next iteration. This makes the cache ineffective and increases the query load on the database. It can negatively affect performance, usually translating into an increase in the latency of metric data ingest.TracesWhen trace data is ingested, spans are translated into the Promscale span schema, which has individual tables for spans and their associated events and links. Resource, span, event, and link attributes are stored in a separate tag table, and their tag IDs are referenced in the span, event, and link tables.Tracing uses multiple caches, but the most relevant one for ingest is the tag cache because of its potential cardinality and size. The tag cache is where the resource, span, link, and event attribute names and values are cached. As new spans, links or events are ingested, and if new attribute names or values are found, they are inserted in the database and cached together with their IDs in the tag cache.When existing attribute names or values are found, their corresponding IDs in the database are retrieved from the cache. This cache behaves in a similar way to the metric series cache. It automatically grows to hold more attribute names and values up to a certain limit. At this point, new attribute names and values cause older attribute names and values present in the cache to be evicted. If the cache is too small to hold all the active attributes, the constant cache evictions and subsequent database queries will cause performance degradation.Data reads with PromQLPromscale is built on PostgreSQL, but it reuses parts of the Prometheus PromQL query evaluation code, giving it 100 % PromQL compliance.To process PromQL queries, the most straightforward approach for Promscale would be to retrieve all the matching metric series data points (Prometheus calls them samples) from the database and let the PromQL query evaluation code process them. But for queries that return a lot of data points to be processed, this would require a lot of memory and CPU, which may lead to long query executions and possibly even failures.In Promscale, PromQL queries may be used not only by dashboards but also by alerting and recording rules, so it is essential to ensure that they complete successfully and run fast.To speed up query execution and make the process more efficient, the Promscale Connector parses the PromQL query and translates it into “query pushdowns.” That means it runs parts of the PromQL query directly inside the database via SQL, leveraging TimescaleDB’s time series capabilities. The Promscale extension also provides additional functions to help map a higher percentage of the PromQL queries to SQL.Data managementThe Promscale database automatically handles compression and retention policies. It does so by regularly running background jobs. By default, two background jobs run every 30 minutes.For high data ingest volume, the database may run behind on applying compression and background jobs, resulting in background jobs taking a very long time to complete and increasing disk usage.The most common ways to resolve this are the following:Reduce the ingest rate by filtering unneeded seriesConfigure additional background jobs if your compute has more CPUs availableIncrease the amount of compute resources allocated to the databaseAlerts, Runbooks, and Dashboards to Fix These IssuesOnce we identified the most likely potential problems that users could find, we started building our set ofout-of-the-box alerts,runbooks, anddashboardsto help our users ensure everything was working smoothly—from ingesting data to running PromQL queries and running maintenance tasks for data compression and retention.When creating the alerting rules, we followed these design principles:Alerting rules should be symptom-based (e.g., “ingest latency increasing,” which alerts you onactualperformance degradation that users will experience vs. “high CPU consumption,” which may or may not negatively impact the experience). The metrics used to trigger those alerts should explain the cause.Alerts should be actionable: they should help you fix the issue immediately. For this reason, we decided to create runbooks for each one of the alerts.Things should stay simple: we should avoid too many alerting rules, leading toalerting fatigueThe resulting alerts live inthis YAML file. As you can see, if you browse the code, we grouped the alerting rules into several categories, aligned with the areas more prone to causing potential performance bottlenecks. To help you visualize everything, we also built aGrafana dashboardincluding several panels associated with these alerts.Such categories are presented in the paragraphs below.Promscale down (1 alert)This alert checks if a Promscale instance is running. The runbook associated with this alert liveshere.Grafana dashboard monitoring Promscale: general overviewIngest (4 alerts)This set of alerts checks for high latency or error rate in the ingest of telemetry data. They use thepromscale_ingest_requests_totalandpromscale_ingest_duration_seconds_bucketmetrics, which are available for metrics and traces via the type label. The former also includes a code label that you can use to identify requests that returned an error.The runbooks related to these alerts livehereandhere.Grafana dashboard monitoring the ingest of telemetry data into PromscaleQuery (4 alerts)These alerts check for high latency or error rate in PromQL queries. In a similar vein to ingest metrics, they usepromscale_query_requests_totalandpromscale_query_duration_seconds_bucketmetrics with the same labels as the ones for ingesting.The runbooks associated with these alerts livehereandhere.Grafana dashboard monitoring the performance of PromQL queries in PromscaleCache (2 alerts)These alerts check if the cache is large enough to avoid evictions of active items. This is monitored via thepromscale_cache_query_hits_totalandpromscale_cache_evictions_totalmetrics, which also have a type label to separately track issues associated with the metric and trace caches.The runbooks related to these alerts livehereandhere.Grafana dashboard monitoring the cache in PromscaleDatabase connection (2 alerts)These alerts check for high latency or error rate in the connection between the Promscale Connector and the Promscale Database. They leverage several database metrics:promscale_database_requests_errors_totalpromscale_database_requests_totalpromscale_database_requests_duration_bucketpromscale_database_requests_duration_countThe runbooks associated with these alerts livehereandhere.Promscale database (4 alerts)Lastly, these alerts look at potential issues with the Promscale database regarding health checks, compression, and retention jobs. They monitor the following metrics:promscale_database_healthchecks_errorspromscale_database_healthchecks_totalpromscale_sql_database_worker_maintenance_job_start_timestamp_secondspromscale_sql_database_worker_maintenance_job_failedpromscale_sql_database_chunks_countpromscale_sql_database_chunks_compressed_countpromscale_sql_database_metric_countpromscale_sql_database_compression_statusThe runbooks associated with these alerts livehere,here,here, andhere.Grafana dashboard monitoring the Promscale DatabaseAll these alerts are based on the metrics exposed by Promscale’s Prometheus-compliant/metricsendpoint, which runs on port 9201 by default.In total, we built 17 alerts and 13 runbooks.Observing the Observer: How to Get StartedThis set of tools is freely available with thelatest Promscale release. And all the information on how to start using it lives in our documentation:Learn how to configure alerting rules in Promscale, usingthis YAML configuration fileto set up this particular set of “observing the observer” alerts.All the runbooks associated with the alerts can be foundin this GitHub repo.Lastly,import the Grafana dashboard into your own instance.If you are still not using Promscale, you can install ithere(it’s 100 % free) or get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.And if you are using Kubernetes, an even more convenient option is to install Promscale usingtobs, a tool that allows you to install a complete observability stack in a few minutes. This set of alerting rules and dashboards have been directly integrated with tobs, so if you use tobs to deploy Promscale, they will be automatically deployed as well.#AlwaysBeLaunchingIngest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-we-built-alert-rules-runbooks-and-dashboards-to-observe-our-observability-tool/
2021-02-18T15:00:45.000Z,Introducing Tobs: Deploy a Full Observability Suite for Kubernetes in Two Minutes,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Tobs will deploy a full observability suite into your Kubernetes cluster with only a single command, so you can start collecting, analyzing, and visualizing all of your metrics in only a few minutes. While the set of components and their configuration is highly opinionated by default, Tobs is also highly extensible and configurable. This makes it easy to get started, but also gives you the flexibility to make the suite your own.Today’s dynamic microservices-based architectures require advanced monitoring and observability capabilities. Unlike the monolithic systems of the past, today’s systems are composed of many interconnected components orchestrated through a significant degree of automation. Troubleshooting problems with modern systems requires shifting through the monitoring data of many individual components and piecing together enough information to conduct root cause analysis.Observability tools help troubleshoot these problems and come in two main flavors: proprietary and open-source. Proprietary solutions such as DataDog and Splunk offer easy turn-key solutions for monitoring and observability. It is easy to get started with proprietary tools, and they may cover a great deal of the need for observability. But they tend to be rigid and inflexible, and users don’t want to be at the mercy of a SaaS provider's changing pricing structure.Increasingly, users opt for open-source solutions instead. Open-source observability involves bringing together multiple interconnected tools, each of which tend to do one thing well — e.g., TimescaleDB for long-term data storage, Prometheus for collecting metrics, Fluentd for collecting logs, Grafana for visualization, Jaeger for tracing, etc. From a software engineering perspective, loosely coupled and highly customizable tools provide the flexibility we need.However, setting up observability suites can be intimidating for developers. We have often seen new users get lost and overwhelmed when trying to implement observability into their systems for the first time. Luckily, infrastructure automation tools like Helm, Ansible, and Puppet can help, but they don’t help developers decide which tools to use or how to configure them together.This is why we created Tobs, an open-source tool that leverages Helm and enables users to deploy an observability suite into any Kubernetes cluster with a simple command line instruction.For example, the following command will install and configureTimescaleDB, Promscale, Prometheus, Grafana, PromLens, Kube-State-Metrics, and Prometheus-Node-Exporter:tobs installTobs allows users to get started with observability more easily: previously users had to first decide which tools they needed, how to set up and configure every tool individually, figure out how to connect and secure them, etc. — all before collecting any data or seeing any benefits.With the ability to deploy an entire suite at once with a few commands, users get the power of several curated tools working together to collect and analyze their cluster right away. Users can then start seeing the benefits quickly and can customize their suite as their needs evolve. And, because modern systems emit relentless streams of observability data, we built Tobs around the rock-solid foundation of TimescaleDB, a petabyte-scale relational database for Prometheus data.Currently, Tobs can install the following tools to collect, store, and visualize metrics:TimescaleDBstores and analyzes observability data over the long term.Promscaleprovides the power of PromQL for data stored in TimescaleDB.Prometheuscollects metric data across your cluster.Grafanavisualizes your data through customizable graphs and dashboards.PromLenshelps you build the PromQL queries you need to understand your system.Kube-State-Metricsexposes metrics about the Kubernetes cluster itself.Prometheus-Node-Exporterexposes metric data for nodes in your Kubernetes cluster.The open-source components that make up TobsTobs is a one-stop-shop for all your Kubernetes monitoring needs. We aim to expand the suite with support for logs and traces in the future (and we alwayswelcome community assistancewith adding more components).Because we're developers ourselves, we wanted to make Tobs as easy as possible to operate, while still allowing full customization. We recommend using the CLI, which makes it easy to install, upgrade, customize, and maintain the suite with just a few commands. For example, you can install the entire suite in two minutes (really). If you need to more tightly integrate with a more complex Helm setup, you can use our Helm chart without the CLI as a sub-chart.A Tobs deployment will install the components listed above, connect, and secure them. By leveraging Prometheus service discovery, this deployment will then discover the components in your Kubernetes cluster that are emitting observability data and will start automatically collecting and storing it.Download Tobstoday from our GitHub repositoryRead the Tobs docsContribute to the Tobs communityand provide your input to the projectRead on for more information about how to use Tobs and the kinds of problems it can solve for you.Diving into TobsTobs is a CLI tool designed to install all the observability stack components you need for monitoring in your Kubernetes cluster and provide complete lifecycle support for your monitoring stack. It abstracts all the actions for the observability stack with a single command.You’ve already seen how easy it is to use Tobs to install a monitoring stack:tobs installThis will installTimescaleDB, Promscale, Prometheus, Grafana, PromLens, Kube-State-Metrics, and Prometheus-Node-Exporter.All the components deployed will be configured to connect with the other components. Also, Kubernetes dashboards are pre-configured in the Grafana UI. You can visualize all the observability data you can obtain from your Kubernetes cluster.UpgradingTo upgrade all the components in Tobs, simply run:tobs upgradeForwarding PortsTo access the TimescaleDB, Prometheus, Promscale, PromLens, or Grafana components running in the cluster on your local machine, you can port-forward Tobs.tobs port-forwardThen, simply access the component on its port on localhost.Resetting passwordsTobs allows you to reset passwords for various components. For example, to reset the password for Grafana, you run:tobs grafana change-password <new password>Configuring Metric RetentionTobs allows you to set retention policies on a global basis and per metric basis. For example, to configure the retention policy to 2 days forgo_threadsmetric, you run:tobs metrics retention set go_threads 2Volume ExpansionTobs offers you an easy way to expand persistent volumes claims (PVCs) for TimescaleDB and Prometheus. For example, to expand Prometheus storage and TimescaleDB storage, you can run:tobs volume expand --timescaleDB-storage 175Gi --timescaleDB-wal 25Gi --prometheus-storage 15GiIntegrating Tobs with an external TimescaleDBYou can connect Tobs to an external TimescaleDB (i.e., to an existing or external instance of TimescaleDB outside the k8s cluster) by providing the DB URI.This will skip deploying of TimescaleDB during the Tobs installation and connects the rest of the stack to the provided DB URI.tobs install --external-db-uri postgres://some_user:[email protected]:5432/tsdb?sslmode=preferComponents installed by TobsLet’s dive into each component that Tobs installs and configures on your behalf.TimescaleDBWe use TimescaleDB for long-term storage of metric data. Long-term storage provides the ability to perform post hoc analysis on metric data over long periods of time. Such data analysis can be used for capacity planning, identifying slow-moving regressions, trend analysis, auditing, and more.We picked TimescaleDB as opposed to other systems because it is unique in the ability to perform analytics using SQL. This allows data to be used for much richer analysis than other stores. TimescaleDB also supports high cardinality and is built on top of PostgreSQL, ensuring good reliability and durability of data as well as support for a wide array of high-availability options.Tobs also stores Grafana’s configuration data in TimescaleDB. This allows the Grafana deployment itself to be stateless, easing backup and reliability concerns.PromscaleWhen deploying TimescaleDB as long-term storage, Promscale provides the translation layer between Prometheus and the database. In particular, it allows the Prometheus server to store and retrieve metrics from TimescaleDB, and allows users to use PromQL on Promscale and Prometheus. (Plainly stated, Promscale is the obvious choice when connecting Prometheus and TimescaleDB.)Learn more about Promscale, andread our blog postfor more information.PrometheusPrometheus is an open-source systems monitoring and altering stack. It has become the de-facto standard in metric monitoring and is the basis of standards such as OpenMetrics. It allows you to monitor and understand how your infrastructure and applications are performing. Service discovery allows Prometheus to automagically discover components within your Kubernetes cluster that are already emitting metrics.Learn more about Prometheus.GrafanaGrafana is a popular visualization tool for creating and view rich dashboards based on metrics.To help users gain insights into their cluster right away and see the value, Tobs deploys Grafana with pre-built dashboards to monitor Kubernetes.Learn more about Grafana.PromLensA tool to help users build PromQL queries with ease. PromLens is a PromQL query builder that helps you build, understand, and fix your queries much more effectively.As with any query language, PromQL can be challenging to learn. PromLens makes it easier and is thus an invaluable tool for users who are new to Prometheus and observability.Learn more about PromLens.Kube-State-MetricsKube-state-metrics exports the metrics related to Kubernetes resources, e.g., the status and count of Kubernetes resources, with visibility of the desired resources and the current resources, as well as the trends in your cluster.Learn more about Kube-state-metrics.Node-ExporterNode-Exporter is deployed to export node-related metrics (e.g.d, CPU, memory) from the Kubernetes cluster.Learn more about Node-Exporter.Missing something?Is your favorite tool not included yet?Create an issue and let us know, or better yet,submit a pull request.ConclusionObservability is increasingly critical in today’s world of complex microservices architecture.Proprietary solutions are easy to get started, but can be inflexible and costly in the long run. Open-source solutions are complex to configure and start with but can be fully customized and cost-effective once implemented.We built Tobs to make open-source systems accessible to everyone. Tobs is an easy-to-use, open-source tool for deploying an observability suite into any Kubernetes cluster. With a simple command-line instruction, you can get up and running in under two minutes.By reducing the startup time, we hope to spark even greater adoption of open-source observability solutions.Next StepsDownload Tobstoday from our GitHub repositoryRead the Tobs docsContribute to the Tobs communityand provide your input to the projectLet us knowabout other tools you’re interested in seeing added to TobsIngest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/introducing-tobs-deploy-a-full-observability-suite-for-kubernetes-in-two-minutes/
2023-01-13T14:30:58.000Z,How to Successfully Migrate From Jaeger to OpenTelemetry Tracing,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.This blog post is based on “Tips and Tricks to Successfully Migrate From Jaeger to OpenTelemetry,” a presentation by Timescale’s Observability product manager, Vineeth Pothulapati, at KubeCon|CloudNativeCon North America 2022.My daily work as a product manager involves developing and advancing observability products. So, it wasn’t a surprise that I wanted to dig into a topic I spent so much time reflecting on during my talk at the 2022 KubeCon|CloudNativeCon North America.As part of the Observability Team at Timescale, I’m mainly focused on developing products such asPromscale (our unified metric and trace storage for Prometheus, Jaeger, and OpenTelemetry built on PostgreSQL and TimescaleDB)andTobs (our Observability stack for Kubernetes).On the side, I’m also a maintainer of theOpenTelemetry Operator. By the way, if you’re using it, I’d love to hear about your experience—drop me a line in theTimescale Community Slack(@Vineeth)!WhenJaeger(a distributed tracing tool many of us have been using and loving for a while now) announced its client libraries’ end-of-life earlier last year, I knew I wanted to present an alternative you could migrate to at KubeCon. The Jaeger community and its maintainers have supported and advocated for the OpenTelemetry SDK, encouraging me to focus on migrating from one tracing tool to the other.So if, like myself, you’re using Jaeger and have to move from its client libraries to theOpenTelemetrySDK, check out my presentation below. I’ll summarize the main points in this blog post, but for a complete picture—including an OpenTracing API demo—I recommend you watch the video.Migrating From Jaeger to OpenTelemetry Tracing: PrerequisitesI divided my presentation agenda into the following topics:PrerequisitesWhy migrate?Jaeger and OpenTelemetry architectureLevels of migrationJaeger and OpenTelemetry boundariesAs you may have guessed, let’s start with the prerequisites.While OpenTelemetry (which I often abbreviate to OTel during the presentation and in this post) also supports metrics and logs alongside traces, my talk focused on traces. And above all, I didn’t intend to push you to migrate or sell anything. I just wanted to share how you can mix and match things and the upsides of migration.But for that, I had first to explain all the components involved.Tracing componentsThere are multiple components in the tracing world, both in Jaeger and OpenTelemetry. This means the instrumentation layer usually comprises an API and SDK.We use the OpenTracing API and Jaeger client libraries as the SDK in Jaeger. The agent/collector is all Jaeger and offers some native storage options within the collector. It also has a visualization layer called Jaeger query, which I discussed in the presentation, and helps you visualize the traces even if you’re using OTel.Source: Jaeger documentationIn OpenTelemetry, there’s only an instrumentation layer and the collector layer. So, while Jaeger goes all the way from instrumentation to storage and visualization, OTel is purpose-built for instrumentation and collection only.Source: OpenTelemetry documentationWhy Migrate?The OpenTelemetry project was precisely announced in 2019 at KubeCon (San Diego). Since then, it has been evolving and expanding into different observability signals and adding new capabilities. Let’s discuss what those capabilities are and why a migration would make sense.I already mentioned the first reason: Jaeger stopped supporting its client libraries in favor of OTel SDKs. The second is that OpenTelemetry is a new instrumentation and data collection standard. It takes some of the industry’s best practices that were part of open tracing and open census and adds new capabilities that modern cloud-native applications need.That makes OTel’s collector layer incredibly rich: it allows you to configure different sources and destinations to ship your data while supporting auto-instrumentation—ensuring no code changes are involved. It also offers processors to enrich the data while it’s received and exported.Levels of MigrationBefore we get into the two levels of migration—the instrumentation layer and the collector layer—let me quickly walk you through the Jaeger and OTel architectures.You can use Jaeger to instrument your application, with spans being pushed to the Jaeger agent/collector. From there, you’ll find the storage backend and the user interface (UI). This is the complete ecosystem and components involved in the Jaeger architecture. The spark jobs are optional: you can run them if you need to.Now for the OpenTelemetry architecture: we cannot see the UI layer or the storage layer. It's all about the instrumentation and the data processing pipeline, which is OpenTelemetry’s Collector, which can be run as an agent based on the machine it’s part of.People often need clarification on the agent and the collector. The agent is run within the whole store as a sidecar to the application. The collector acts as a centralized processing pipeline where your applications can directly send the spans to the collector, which the agent can perform.Instrumentation LayerWhen considering a migration from Jaeger to the OpenTelemetry SDK, the first level is the instrumentation layer. As mentioned, there is an API and an SDK. The API contains a tracer, the API itself, the context API, and the meter APIs for metrics. In the SDK, you will find a propagator, a span processor, and an aggregator. So these are the functionalities that the API and SDK offer you during instrumentation.You can complete the migration in the instrumentation layer in two ways:1.OpenTelemetry shimThe shim is a library that facilitates the migration between OpenTracing and OpenTelemetry. It consists of a set of classes that implement the OpenTracing API while still using the OTel constructs behind the scenes.You’ll find a great explanationin this blog post written by Juraci Paixão Kröhling, in which he uses the Java application as a demo. With minimal code changes, it will hardly take five minutes to migrate by swapping the dependencies and the imports.If you have less bandwidth and want to use client-based sampling in OpenTelemetry or simply want to try the OpenTelemetry SDK for some reason, you can definitely start with shim.2.Complete re-instrumentationA complete re-instrumentation is the second way to help you get on the OpenTelemetry SDK, which offers OTel as a package. This means you will get all the capabilities from scratch, from the code to semantics. In the future, you can also expand your OTel instrumentation into metrics and logs and easily integrate it with auto-instrumented applications.So if you want to do auto-instrumentation for a few applications and others for which you need more granular detail, auto-instrumentation will give you higher-level traces. With manual instrumentation, you will have more flexibility over what you want to capture and what you want to measure.Demo timeMy re-instrumentation demo is a clone of anOpenTracing tutorial authored by Yuri Shkuroand lives on GitHub—it will help you understand how instrumentation works.I took the same application and showed you both the Jaeger and OTel instrumentations. The demo is available in thisGitHub repository. It aims to show you how simple it is to run and use the Jaeger and OpenTelemetry instrumentation alongside one another.The video starts at demo timeWhat is the impact after you migrate?Improved tracer implementationSwitch to the OpenTelemetry SDK while continuing to use your existing OpenTracing instrumentationImproved performanceAccess to OpenTelemetry’s framework pluginsCollector LayerLet’s now move on to the basics of the second migration level: the collector layer, which will allow you to migrate from the Jaeger to the OpenTelemetry Collector without touching the code or disturbing your applications.The anatomy of the OpenTelemetry CollectorWhen deploying the Jaeger Collector, you can simply add the OpenTelemetry Collector into your existing architecture without making OTel code changes to your applications. The OTel Collector can receive data from Jaeger and other different formats.Check out my talk to see the differences between both collectors and how to configure them.What is the impact after you migrate?Migrating the collector moves the complete data processing and storage backend away from Jaeger.You can configure pipelines to receive and send data from multiple sources to destinations.This is a vendor-neutral processing system: you can seamlessly migrate from one vendor to another by changing the collector configuration.Leverage OpenTelemetry’s rich data collection capabilities with support for a wide range of receivers, processors, and exporters.In sum, why should you use OTel in Jaeger?You can leverage the best in both worlds by using both collectors.As a project, Jaeger is becoming a tracing platform that offers storage, querying, and visualization of traces.Jaeger offers native support for Promscale, Cassandra, Elasticsearch, Badger, and in-memory storage systems.Jaeger exposes a gRPC-based remote write integration, allowing you to plug the desired backend to store traces, such as Promscale.Jaeger and OpenTelemetry BoundariesFinally, one of the most crucial aspects of the migration is how you will keep querying and visualizing your traces. If you move completely to the OpenTelemetry collector, there is no path to visualize traces using the Jaeger UI unless the storage backend offers support for querying and visualizing traces using the Jaeger query component, like Promscale.The OTel project is all about collecting, instrumenting, and collecting data. Whereas with Jaeger, you can use the Jaeger UI to visualize the data. So if you are moving from Jaeger to OpenTelemetry,  you should integrate the Jaeger UI to fully query your traces.If you want to learn more about how the OpenTelemetry Collector interacts with Jaeger to aggregate traces into Prometheus metrics and graph them inside Jaeger’s UI,check out this blog post by Timescale’s own Mathis Van Eetvelde.In fact, these are some of the Jaeger-OTel boundaries: OTel is all about the API, SDK, and the OpenTelemetry Collector. Jaeger is the query, mature native storage backend. So in the future, I see Jaeger evolving into a platform for traces, whereas OTel will be more like an instrumentation and collection pipeline for all the observability data.Long-Term Storage for Jaeger and OpenTelemetry TracesIf you are looking to store and correlate metrics and traces,we recommend using PostgreSQL for Jaeger with Promscale.Promscale is a unified metric and trace storage for Prometheus, Jaeger, and OpenTelemetry built on PostgreSQL and TimescaleDB. With Promscale, you get a centralized and reliable long-term storage for your metrics and traces that offers the following:Full Jaeger support: passes all the Jaeger storage certification tests, has native support for OpenTelemetry and can be used as the metric storage backend for Jaeger’s Service Performance Monitoring (SPM) feature.First-class Prometheus support: fully PromQL-compliant and support for PromQL alerts and recording rules, exemplars, Prometheus high availability, and multi-tenancy.Flexible storage: configurable downsampling and retention policies, including per-metric retention, data backfilling and deletion, and full support for both PromQL and SQL.Rock-solid foundation: built on the maturity of PostgreSQL and TimescaleDB with millions of instances worldwide. A trusted system offering scalability, high availability, replication, and data integrity.Want to try it out? The easiest way to get started is tosign up for Timescale (create a free 30-day account, no credit card required).Self-hostingis also available for free.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-successfully-migrate-from-jaeger-to-opentelemetry-tracing/
2022-07-07T13:55:58.000Z,Prometheus vs. OpenTelemetry Metrics: A Complete Guide,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Welcome to the third and final post of our series about metrics. First,we deep-dived into the four types of Prometheus metrics;then, weexamined how metrics work in OpenTelemetry; and finally, we are now putting the two together—explaining the differences, similarities, and integration between the metrics in both systems.We decided to round off the series by comparing the metrics in both tools because we believe this is a choice you’ll need to make sooner rather than later. While Prometheus has been the current standard for monitoring your systems, OpenTelemetry is quickly gaining ground, especially in the cloud-native landscape, which was traditionally Prometheus’ stronghold. OpenTelemetry’s promise of creating a unified standard among traces, logs, and metrics with enough flexibility to model and interact with other approaches is tempting many developers.After working with both frameworks, our goal with this blog post is to compare the two, show you how to transform one into the other, and share some thoughts on which may be more appropriate, depending on your use case.Metrics in Prometheus vs. OpenTelemetry: Common GroundBoth systems allow you to collect and transform metrics (although Open Telemetry is much more flexible as a transformation pipeline). However, looking back at the previous articles, we need to remember an important distinction: Prometheus is an Observability tool (including collection, storage, and query) that uses a metric model designed to suit its own needs.On the other hand, the metrics component of OpenTelemetry translates from many different data models into one single framework (providing collection with no storage or query). These core differences reflect the systems’ complexity—Prometheus is a straightforward model which is often exposed in text, while OpenTelemetry is a more intricate series of three models which uses a binary format for transmission.Both systems also allow code instrumentation via an SDK, but OpenTelemetry also focuses on supporting automatic instrumentation, which does not add any code to applications (where possible).So what’s the overlap between the metrics you can create? Essentially, OpenTelemetry allows the representation of all Prometheus metric types (counters, gauges, summaries, and histograms). Still, Prometheus can’t represent some configurations of OpenTelemetry metrics, including delta representations and exponential histograms (although these will be added to Prometheus soon), as well as integer values.In other words, Prometheus metrics are a strict subset of OpenTelemetry metrics.Main DifferencesWhile there are some differences in how the internal models work (read on for more information), the practical differences between the two from a developer’s point of view are more to do with the ecosystem.Prometheus provides metric collection, storage, and query. It generally gathers metrics via a simple scraping system that pulls data from hosts. The Prometheus database stores that data, which you can then query with the Prometheus query language, PromQL. The Prometheus database can handle a lot of data, but it’s not officially meant to be a long-term storage solution, so data is often sent to another storage solution—likePromscale— after some time but still read back via PromQL.OpenTelemetry has a much smaller scope. It collects metrics (as well as traces and logs) using a consolidated API via push or pull, potentially transforms them, and sends them onward to other systems for storage or query. By only focusing on the parts of Observability which applications interact with, OpenTelemetry is decoupling the creation of signals from the operational concerns of storing them and querying them. Ironically, this means OpenTelemetry metrics often end up back in Prometheus or a Prometheus-compatible system.When we are looking at actual metric types, there are several differences:OpenTelemetry can represent metrics as deltas rather than as cumulative, storing the difference between each data point rather than the cumulative sum. Prometheus does not allow this by design (although you can calculate the values at query time). This isn’t the default in OpenTelemetry and would mainly be used for metrics that would only ever be expressed as rates.OpenTelemetry also allows metric values to be integers rather than floating-point numbers, whichPrometheus can not express.OpenTelemetry can attach some extra metadata to histograms, allowing you to track the maximum and minimum values.Finally, OpenTelemetry has an exponential histogram aggregation type (which uses a formula and a scale to calculate bucket sizings). Prometheus can not represent this today, but does have a fully compatible metric type in the works!Choosing Between the TwoIf you don't already have an investment in one of the two technologies, the choice between Prometheus and OpenTelemetry might boil down to four questions:Are you planning on capturing traces, logs, and metrics? If so, OpenTelemetry will allow you to use the same libraries to instrument across all three signal types, which is a significant benefit. You can even send all three signals to the same backend and use a single language to query across them (for example, Promscale and SQL).Do you value stability and battle-tested systems? If so, Prometheus might be the correct answer for a few more years as OpenTelemetry gets production exposure.Would you like to use a multi-step routing and transformation pipeline? If so, perhaps OpenTelemetry might be worth a look.Do you want to be able to stay as flexible as possible? Then OpenTelemetry is for you, as it doesn’t implement any storage or query, giving you maximum flexibility.Most organizations will likely mix both standards: Prometheus for infrastructure monitoring, making use of the much more mature ecosystem of integration to extract metrics from hundreds of components, and OpenTelemetry for services that have been developed. Many engineers will probably use Prometheus as a backend to store both Prometheus and OpenTelemetry metrics, and will need to ensure the OpenTelemetry metrics they produce are compatible with Prometheus.In practice, this means opting for the cumulative aggregation temporality and only using OpenTelemetry metric types supported by Prometheus (leaving aside OpenTelemetry exponential histograms until Prometheus adds support for these).Converting Between Prometheus and OpenTelemetryIf you want to mix and match the standards, then the good news is that OpenTelemetry provides theOpenTelemetry Collector, which can help with moving in both directions (even casting between types if needed in some cases).The OpenTelemetry Collector is pluggable, allowing both receivers and exporter components to be enabled using a config file at runtime. We will be using thecontribpackage that includes many receivers and exporters. You can download the appropriate binary from theGitHub Release pages. Alternatively, if you’re running the collector in production, you can also compile a version containing just the components you need using theOpenTelemetry Collector Builder.For the examples in the following sections, we are running the collector with the followingconfigfile which is saved asconfig.yaml:receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: demo
          scrape_interval: 15s
          static_configs:
            - targets: ['localhost:9090']


exporters:
  prometheus:
     endpoint: ""0.0.0.0:1234""
  logging:
    loglevel: debug

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: []
      exporters: [logging,prometheus]Then we start the collector with:otelcol –config config.yamlThe collector will use the Prometheus receiver to try to scrape a Prometheus service athttp://localhost:9100. If you need something to test with, you could start a localnode_exporterthat uses this port. As data is scraped from this service, you will see it show up as log output from the collector, and it will also be available from the Prometheus exporter endpoint, which the collector will run onhttp://localhost:1234.The Prometheus Remote Write Exporter is also an option, but it’s more limited in scope at this stage, only being able to handle cumulative counters and gauges.Diagram illustrating how Prometheus interacts with the OpenTelemetry CollectorIf we look back to our previous post onPrometheus metrics, we covered the four main metric types: counters, gauges, histograms, and summaries. In the Prometheus context, a counter is monotonic (continuously increasing), whereas a gauge is not (it can go up and down).If you’re really on point, you’ll also remember that the difference between a histogram and a summary is that a summary reports quantiles and doesn’t require as much client computation. A histogram, on the other hand, is more flexible, providing the raw bucket widths and counts.OpenTelemetry, in contrast, has five metric types: sums, gauges, summaries, histograms, and exponential histograms. If you’ve been following along with thehow metrics work in OpenTelemetrypost, you will have a question at this stage—are these different types from what we have previously seen? The answer to this question lies in the three OpenTelemetry models.If you are a developer creating OpenTelemetry metrics, you deal with the Event model, which is then translated (for transmission) to the OpenTelemetry Protocol (OTLP) Stream Model by the OpenTelemetry SDK. The types we are referencing here are part of this model, which the Prometheus receiver translates directly into.Luckily, these new metric types are self-explanatory and map directly onto the Prometheus metric types (summary is implemented only for Prometheus compatibility, and you won’t see it used elsewhere). The exception is the exponential histogram, which can’t be converted to Prometheus today (but will be able to be converted in the future). The diagram below describes how the mappings work in each direction.Mapping OpenTelemetry metrics and Prometheus metricsOpenTelemetry promises lossless conversions toandfrom Prometheus metrics, giving users the ability to convert as they need without worrying about data loss.Converting From Prometheus to OpenTelemetryLet’s explore how the Prometheus to OpenTelemetry conversions work by looking at examples of Prometheus scrapes and OpenTelemetry metrics. While OpenTelemetry doesn’t have a text representation like Prometheus, we can use the Logging Exporter to emit a text dump of the metrics captured.OpenTelemetry will extract some information from the scrape itself and store this, producing the following output, which defines Resource labels that will be attached to all metrics.Resource SchemaURL: 
Resource labels:
     -> service.name: STRING(node-exporter)
     -> service.instance.id: STRING(127.0.0.1:9100)
     -> net.host.port: STRING(9100)
     -> http.scheme: STRING(http)CountersIf we take the following Prometheus scrape data and point the collector at it:# HELP http_requests_total Total HTTP requests served
# TYPE http_requests_total counter
http_requests_total{method=""post"",code=""200""} 1028
http_requests_total{method=""post"",code=""400""}    5Then the collector would output the following metric from the Logging Exporter.Metric #0
Descriptor:
     -> Name: http_requests_total
     -> Description: Total HTTP requests served
     -> Unit: 
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
NumberDataPoints #0
Data point attributes:
     -> code: STRING(200)
     -> method: STRING(post)
StartTimestamp: 2022-06-16 11:49:22.117 +0000 UTC
Timestamp: 2022-06-16 11:49:22.117 +0000 UTC
Value: 1028.000000
NumberDataPoints #1
Data point attributes:
     -> code: STRING(400)
     -> method: STRING(post)
StartTimestamp: 2022-06-16 11:49:22.117 +0000 UTC
Timestamp: 2022-06-16 11:49:22.117 +0000 UTC
Value: 5.000000We can see that the metric type (DataType) isSumand theAggregationTemporalityisCumulative(the only aggregation that Prometheus supports). There are two timestamps per data point to track counter resets:Timestampis the time of the recording, andStartTimestampis either the time the first sample was received or the time of the last counter reset. There is noUnitspecified. This is because Prometheus specifies units by including them as part of the textual metric name, which can’t be accurately decoded by the OpenTelemetry Collector. Interestingly, using the compatibleOpenMetricsformat to add a unit does not work either.GaugesIf we take a Prometheus gauge and scrape it:# HELP node_filesystem_avail_bytes Available bytes in filesystems
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{method=""/data"",fstype=""ext4""} 250294We would see the following output from the collector.Metric #0
Descriptor:
     -> Name: node_filesystem_avail_bytes
     -> Description: Available bytes in filesystems
     -> Unit: 
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> fstype: STRING(ext4)
     -> method: STRING(/data)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2022-06-23 07:42:07.117 +0000 UTC
Value: 250294.000000Here, theDataTypeis set toGauge.Gaugeis the default metric type OpenTelemetry will convert into, so the lack of a # TYPE line in the Prometheus scrape data will result in a gauge. Prometheus doesn’t actually use the type information itself (it doesn’t differentiate between counters and gauges internally), so some exporters will forgo the two comment lines to make the scrape more efficient. This would result in all OpenTelemetry metrics being gauges.HistogramsA Prometheus histogram which was scraped as:# HELP http_request_duration_seconds Histogram of latencies for HTTP requests.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{handler=""/"",le=""0.1""} 25547
http_request_duration_seconds_bucket{handler=""/"",le=""0.2""} 26688
http_request_duration_seconds_bucket{handler=""/"",le=""0.4""} 27760
http_request_duration_seconds_bucket{handler=""/"",le=""1""} 28641
http_request_duration_seconds_bucket{handler=""/"",le=""3""} 28782
http_request_duration_seconds_bucket{handler=""/"",le=""8""} 28844
http_request_duration_seconds_bucket{handler=""/"",le=""20""} 28855
http_request_duration_seconds_bucket{handler=""/"",le=""60""} 28860
http_request_duration_seconds_bucket{handler=""/"",le=""120""} 28860
http_request_duration_seconds_bucket{handler=""/"",le=""+Inf""} 28860
http_request_duration_seconds_sum{handler=""/""} 1863.80491025699
http_request_duration_seconds_count{handler=""/""} 28860Would present in OpenTelemetry through the Logging Exporter as:Metric #0
Descriptor:
     -> Name: prometheus_http_request_duration_seconds
     -> Description: Histogram of latencies for HTTP requests.
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
HistogramDataPoints #0
Data point attributes:
     -> handler: STRING(/)
StartTimestamp: 2022-06-23 07:54:07.117 +0000 UTC
Timestamp: 2022-06-23 07:54:07.117 +0000 UTC
Count: 28860
Sum: 1863.804910
ExplicitBounds #0: 0.100000
ExplicitBounds #1: 0.200000
ExplicitBounds #2: 0.400000
ExplicitBounds #3: 1.000000
ExplicitBounds #4: 3.000000
ExplicitBounds #5: 8.000000
ExplicitBounds #6: 20.000000
ExplicitBounds #7: 60.000000
ExplicitBounds #8: 120.000000
Buckets #0, Count: 25547
Buckets #1, Count: 1141
Buckets #2, Count: 1072
Buckets #3, Count: 881
Buckets #4, Count: 141
Buckets #5, Count: 62
Buckets #6, Count: 11
Buckets #7, Count: 5
Buckets #8, Count: 0
Buckets #9, Count: 0We can see a histogram is created in OpenTelemetry, but one thing we can’t do is include minimum and maximum values (which OpenTelemetry supports, but Prometheus doesn’t).SummariesA Prometheus summary which is scraped as the following:# HELP prometheus_rule_evaluation_duration_seconds The duration for a rule to execute.
# TYPE prometheus_rule_evaluation_duration_seconds summary
prometheus_rule_evaluation_duration_seconds{quantile=""0.5""} 6.4853e-05
prometheus_rule_evaluation_duration_seconds{quantile=""0.9""} 0.00010102
prometheus_rule_evaluation_duration_seconds{quantile=""0.99""} 0.000177367
prometheus_rule_evaluation_duration_seconds_sum 1.623860968846092e+06
prometheus_rule_evaluation_duration_seconds_count 1.112293682e+09Would result in an OpenTelemetry metric which outputs via the Logging Exporter as:Metric #3
Descriptor:
     -> Name: prometheus_rule_evaluation_duration_seconds
     -> Description: The duration for a rule to execute.
     -> Unit: 
     -> DataType: Summary
SummaryDataPoints #0
StartTimestamp: 2022-06-23 07:50:22.117 +0000 UTC
Timestamp: 2022-06-23 07:50:22.117 +0000 UTC
Count: 1112293682
Sum: 1623860.968846
QuantileValue #0: Quantile 0.500000, Value 0.000065
QuantileValue #1: Quantile 0.900000, Value 0.000101
QuantileValue #2: Quantile 0.990000, Value 0.000177We can see that the OpenTelemetrySummarymetric type has been selected here—remember that this was explicitly created for Prometheus integration and should not be used anywhere else. It’s similar to the histogram output but lists quantiles rather than explicit buckets and bucket counts. In this case, it looks like we are losing some precision, but fear not. This is just the Logging Exporter pretty-printing, as we will see in the next section.Converting From OpenTelemetry to PrometheusUsing either the Prometheus Exporter to allow scraping or the Prometheus Remote Write Exporter to push directly to another Prometheus instance, we can transmit metrics from OpenTelemetry to Prometheus. There aren’t any surprises on this side: if the conversion is supported, it happens without any loss of precision.One thing to remember is that there are some configurations of OpenTelemetry metrics that we can’t translate directly into Prometheus metrics because Prometheus has a much more constrained model.Any metrics with anAggregationTemporalityofDELTAwill be converted back intoCUMULATIVEby the Prometheus Exporter (and will be rejected by the Prometheus Remote Write Exporter). The Prometheus Remote Write Exporter will also reject summary and histogram metrics, but these are managed perfectly by the Prometheus Exporter.  OpenTelemetry metrics with an integer value will be converted into float values.For instance, a scrape from the Prometheus Exporter (when all the examples from the above sections have been ingested) would produce the following results. You will see that each value is exactly the same as the input Prometheus value from the previous sections, with no loss of fidelity.​​# HELP http_request_duration_seconds Histogram of latencies for HTTP requests.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""0.1""} 25547
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""0.2""} 26688
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""0.4""} 27760
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""1""} 28641
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""3""} 28782
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""8""} 28844
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""20""} 28855
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""60""} 28860
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""120""} 28860
http_request_duration_seconds_bucket{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter"",le=""+Inf""} 28860
http_request_duration_seconds_sum{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter""} 1863.80491025699
http_request_duration_seconds_count{handler=""/"",instance=""127.0.0.1:9100"",job=""node-exporter""} 28860
# HELP http_requests_total Total HTTP requests served
# TYPE http_requests_total counter
http_requests_total{code=""200"",instance=""127.0.0.1:9100"",job=""node-exporter"",method=""post""} 907
http_requests_total{code=""400"",instance=""127.0.0.1:9100"",job=""node-exporter"",method=""post""} 5
# HELP node_filesystem_avail_bytes Available bytes in filesystems
# TYPE node_filesystem_avail_bytes gauge
node_filesystem_avail_bytes{fstype=""ext4"",instance=""127.0.0.1:9100"",job=""node-exporter"",method=""/data""} 250294
# HELP prometheus_rule_evaluation_duration_seconds The duration for a rule to execute.
# TYPE prometheus_rule_evaluation_duration_seconds summary
prometheus_rule_evaluation_duration_seconds{instance=""127.0.0.1:9100"",job=""node-exporter"",quantile=""0.5""} 6.4853e-05
prometheus_rule_evaluation_duration_seconds{instance=""127.0.0.1:9100"",job=""node-exporter"",quantile=""0.9""} 0.00010102
prometheus_rule_evaluation_duration_seconds{instance=""127.0.0.1:9100"",job=""node-exporter"",quantile=""0.99""} 0.000177367
prometheus_rule_evaluation_duration_seconds_sum{instance=""127.0.0.1:9100"",job=""node-exporter""} 1.623860968846092e+06
prometheus_rule_evaluation_duration_seconds_count{instance=""127.0.0.1:9100"",job=""node-exporter""} 1.112293682e+09
# HELP scrape_duration_seconds Duration of the scrape
# TYPE scrape_duration_seconds gauge
scrape_duration_seconds{instance=""127.0.0.1:9100"",job=""node-exporter""} 0.003231334
# HELP scrape_samples_post_metric_relabeling The number of samples remaining after metric relabeling was applied
# TYPE scrape_samples_post_metric_relabeling gauge
scrape_samples_post_metric_relabeling{instance=""127.0.0.1:9100"",job=""node-exporter""} 20
# HELP scrape_samples_scraped The number of samples the target exposed
# TYPE scrape_samples_scraped gauge
scrape_samples_scraped{instance=""127.0.0.1:9100"",job=""node-exporter""} 20
# HELP scrape_series_added The approximate number of new series in this scrape
# TYPE scrape_series_added gauge
scrape_series_added{instance=""127.0.0.1:9100"",job=""node-exporter""} 20
# HELP up The scraping was successful
# TYPE up gauge
up{instance=""127.0.0.1:9100"",job=""node-exporter""} 1When converting to Prometheus, keep in mind that we only have two options currently: one is using the Prometheus Exporter, which will mean all our data is exposed as a single scrape that won’t scale well if you have a large volume of series. The second option is using the Prometheus Remote Write Exporter, which we expect to scale better but is limited to counters and gauges and won’t performDELTAtoCUMULATIVEconversions (it will drop these metrics).Decision TimeSumming up, Prometheus and OpenTelemetry provide metrics implementations with slightly different angles. While Prometheus is the de facto standard, covering metrics creation, storage, and query, OpenTelemetry is newer, covering only the generation of metrics. Still, it also supports traces and logs with the same SDK.You can mainly convert between the two without any loss of precision—but it pays to know that some metric types will change slightly. For example, all OpenTelemetryDELTAmetrics will be converted toCUMULATIVEbefore export as Prometheus metrics, and Prometheus cannot represent OpenTelemetry exponential histograms until they add support (which will hopefully be soon).Most organizations will likely mix standards, but if you’re wondering which one to adopt, we recommend you weigh your priorities: do you value stability or flexibility? Are you planning to capture traces, logs, and metrics, or are you okay with metrics only?Prometheus will give you a battle-tested system. OpenTelemetry, a more expansive and flexible standard. The final decision depends only on you, but we hope this blog post has given you some helpful clues.And if you are looking for a long-term store for your Prometheus metrics, check outPromscale, the observability backend built on PostgreSQL and TimescaleDB. It seamlessly integrates with Prometheus, with 100% PromQL compliance, multitenancy, and OpenMetrics exemplars support.Get started now withPromscale on Timescale (free 30-day trial, no credit card required)orself-hostfor free.If you have questions, join the #promscale channel in theTimescale Community Slack. You can directly interact with the team building Promscale and other developers interested in Observability. We’re +4,100 and counting in that channel!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/prometheus-vs-opentelemetry-metrics-a-complete-guide/
2021-10-22T13:09:38.000Z,A Different and (Often) Better Way to Downsample Your Prometheus Metrics,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Observability is the ability to measure a system’s state based on the data it generates. To be effective, observability tools first have to be able to ingest data about the system from a wide variety of sources, typically in the form of metrics, traces, logs, and metadata. Second, they must offer powerful, flexible and fast capabilities to analyze and correlate all that data to understand the health and performance of the system and identify issues and areas for improvement.Promscaleis an easy-to-use observability backend, built on top of TimescaleDB. Our vision is to enable engineers to store all their observability data - metrics, traces, logs and metadata - in a single mature and scalable store, and analyze data through a unified and complete SQL interface. Earlier this month, as part of our#AlwaysBeLaunchingseries of monthly launches,we launched support for tracesin Promscale, enabling developers to interrogate their trace data and unlock new insights to help them identify problems and potential optimizations in their microservices environments in a way not possible with other open-source tracing tools. Promscale also supports storing and querying (in SQLandPromQL) metric data fromPrometheus,the de-facto monitoring standard for modern cloud-native environments. Today, we’re excited to introduce a better option for downsampling Prometheus metrics, enabling developers to do accurate and flexible trend analysis on those metrics over long periods of time with high performance and reduced storage costs. Prometheus downsampling leveragescontinuous aggregates, one of the most popular and powerful features of TimescaleDB.Metric monitoring is a key pillar of any observability stack used to operate micro-service-based systems running on Kubernetes. Those systems are very dynamic with frequent deployments, vertical and horizontal autoscaling of pods or automatic provisioning of new nodes in the cluster and are made of lots of individual components. Prometheus is a great fit for those environments thanks to its straightforward auto-discovery mechanism of components to monitor and support for dimensional metrics (i.e., metrics with tags).One thing is certainly true of Prometheus metric data: there is a lot of it. Each component in a Kubernetes-based system (and there are many of them) emits lots of metrics that are collected regularly (every 1 minute by default) and stored by Prometheus. As an example, just the node exporter which is used to monitor hosts emits hundreds of different metric data points (aka samples) for each collection period. Hundreds of thousands (even millions) of samples per second are fairly common for production environments which are expensive to store for long periods of time. Yet, as that data ages, individual samples become less important and we care more about the general trends and aggregates. For example, we would want to have real-time high resolution access to metrics like the number of API requests customers make to our application to detect sudden changes in it as an indication of a potential problem or the need to scale our systems. We would also want to use the same metric to understand how adoption of our API is growing over time in which case we’ll need to query data over long periods of time (a year for example) where resolution is not all that important (one data point per day per customer would be enough). How do we make the query to understand API adoption fast and reduce storage costs by only keeping the data at the resolution we need? We use downsampling.What is downsampling?Downsampling is the ability to reduce the rate of a signal. As a result, the resolution of the data is reduced and also its size. The main reasons why this is done are cost and performance. Storing the data becomes cheaper and querying the data is faster as the size of the data decreases.The easiest form of downsampling is to collect fewer data points. With Prometheus this could be achieved by increasing themetric scraping interval(i.e., decreasing how often Prometheus collects metrics) at the cost of less visibility into metric changes between scrapes. However, as explained above, this is not typically what we want. In observability, the value of data diminishes with its age. We want very high resolution for our more recent data while it’s perfectly fine for old data to have much lower granularity, so it’s cheaper to store and faster to query.A more sophisticated form of downsampling is to summarize and aggregate individual data points, often by bucketing data by time (i.e., hours, days, weeks, etc.). Summarizing data in this way reduces the amount of data that needs to be processed and stored. Therefore it improves the performance of queries for aggregate statistics over longer time spans and allows users to keep information about key features of their data for longer at a reasonable cost. In observability, the individual high-resolution samples are also kept but typically for a much shorter period of time since they are mainly used for troubleshooting issues right after they occur.Downsampling with PrometheusIn the Prometheus ecosystem, downsampling is usually done throughrecording rules. These rules operate on a fairly simple mechanism: on a regular, scheduled basis the rules engine will run a set of user-configured queries on the data that came in since the rule was last run and will write the query results to another configured metric. So if we have a metric calledapi_requests_totalwe can define a metric rule for a new metric calledcustomer:api_requests:rate1dayand configure the rules engine to calculate the daily rate of API requests every hour and write the result to the new metric. The rules file would look like the following:groups:
  - name: daily_stats
    interval: 1h
    rules:
    - record: customer:api_requests:rate1day
      expr: sum by (customer) (rate(api_requests_total[1d]))When it comes to querying the data we will run aPromQLquery against the new aggregated metric. For example to see the evolution in the number of API calls per day by customer we could run the following query:customer:api_requests:rate1dayMost of the Prometheus ecosystem, including Promscale, supports downsampling using recording rules. While recording rules provide an easy-to-use mechanism to speed up long-term queries, they have some important limitations:Data is delayed by the time needed to aggregate it. If the resolution of our recording rule is 1 hour, queries on that metric will not include data since the last aggregation (anywhere between 0 and 60 minutes).They don’t necessarily help with reducing storage costs. Promscale does provide the ability toconfigure a default retention and then per metric retention overridesbut many other storage systems for Prometheus don’t provide the ability to configure different retention policies for different metrics. For example, neither Thanos nor Cortex offer that capability.They specify a particular resolution but if we want to aggregate larger buckets of data we can end up with inaccurate results (e.g. average of averages, aggregating histograms)By default, they are only applied to data points received after the recording rule is created and require an additional manual step to backfill data.Introducing Downsampling with PromscaleToday we are announcing the beta release of an additional downsampling method in Promscale called continuous aggregation that is more timely and accurate than recording rules in many circumstances. Combined, these two methods cover the majority of downsampling use-cases.Read on to learn more about continuous aggregates in Promscale, how to set them up, how to query them and how to decide when to use continuous aggregates and when to use Prometheus recording rules.To get started right away:Install the latest version of Promscale,following the instructions in ourGitHub repository. As a reminder, Promscale is open-source and completely free to use.Join the TimescaleDB Slack community,where you’ll find 7K+ developers and Timescale engineers active in all channels. (The dedicated #promscale channel has 2.5K+ members, so it’s a great place to connect with like-minded community members, ask questions, get advice, and more).And, if these are the types of challenges you’d like to help solve, we are hiring (see all roles)!Benefits of continuous aggregatesPromscale continuous aggregates leverage aTimescaleDB feature of the same nameto have the database manage data materialization and downsampling for us. This mechanism improves on some aspects of recording rules but is not always appropriate. Combined, recording rules and continuous aggregates, cover a large portion of the use-cases we have seen.Continuous aggregates address the following limitations of recording rules:Timeliness. With recording rules, users only see the results of the query once the rules engine has run the materialization but not as soon as data comes in. This might not be such a big deal for 5-minute aggregates (although it could be) but for hourly or daily aggregates it could be a significant limitation. Continuous aggregates have a feature calledreal-time aggregateswhere the database automatically combines the materialized results with a query over the newest not-yet-materialized data to give us an accurate up-to-the-second view of our data.Rollups.Downsampling is defined for particular time-bucket granularities (e.g. 5 minutes). But, when performing analysis, we may want to look at longer aggregates (e.g.  1 hour). With recording rules this is sometimes possible (a minimum of many minimums is the same as the minimum of the samples) but often it isn’t (the median of many medians is not the same as the median of the underlying samples). Continuous aggregates solve this by storing the intermediate state of an aggregate in the materialization, making further rollups possible. Read more about the way we define aggregates in our previousblog post.Query flexibility for retrospective analysis.Once a query for a recording rule is defined, the resulting metric is sufficient to answer only that one query. However, when using continuous aggregates, we can use multi-purpose aggregates. For instance, Timescale’stoolkit extensionhas aggregates that supportpercentilequeries on any percentile, andstatistical aggregatessupporting multiple summary aggregates. The aggregates that we define when we configure the materialization are much more flexible in what data we can derive at query time.Backfilling.Prometheus recording rules only downsample data collected after the recording rule is created. The Prometheus community createda tool to backfill databut requires an additional manual step and has a number oflimitationsthat make it more complex to use on a regular basis or to automate the process. Continuous aggregates automatically downsample all data available including past data so that we can start benefiting from the performance improvements the aggregated metric brings as soon as it is created.Downsampling using continuous aggregates in PromscaleTo downsample data, first we need to have some raw data. In Promscale, each metric is stored in a hypertable which contains the following columns:time column, which stores the timestamp of the reading.value column, which stores the sample reading as a float.series_id column, which stores a foreign key to the table that defines the series (label set) of the reading.This corresponds to thePrometheus data model.Let's imagine we have some metric callednode_memory_MemFree. We can create a continuous aggregate to derive some summary statistics (min, max, average) about the reading on an hourly basis. To do it we will run the following query on the underlying TimescaleDB database which requires using any tool that can connect to PostgreSQL and execute queries likepsql.CREATE MATERIALIZED VIEW node_memfree_1hour
WITH (timescaledb.continuous) AS
  SELECT 
        timezone('UTC', 
          time_bucket('1 hour', time) AT TIME ZONE 'UTC' +'1 hour')  
            as time, 
        series_id,
        min(value) as min, 
        max(value) as max, 
        avg(value) as avg
    FROM prom_data.node_memory_MemFree
    GROUP BY time_bucket('1 hour', time), series_id;Note: we add 1 hour totime_bucketto match the PromQL semantics of representing a bucket with the timestamp at the end of the bucket instead of the start of the bucket.For more information on continuous aggregates and all their options, refer to thedocumentation. This continuous aggregate can now be queried via SQL and we can also make it available to PromQL queries (as a reminder, Promscale is100% PromQL compliant). To do this we have to register it with Promscale as a PromQL metric view:SELECT register_metric_view('public', 'node_memfree_1hour');The first argument is the PostgreSQL schema that contains the continuous aggregate (created in the ""public"" schema by default) and the second is the name of the view. The name of the view becomes the name of the new metric.Now, we can treat this data as a regular metric in both SQL and PromQL.Querying the dataPromscale currently offers two distinct ways of querying data: PromQL and SQL. In this blog post we will primarily use PromQL but we will also show the equivalent SQL queries you would use in a dashboarding tool like Grafana (for more on using Grafana with TimescaleDB and Promscale, see ourGuide to Grafanayoutube series). Note that the raw query results from the PromQL and SQL queries would be formatted differently but they would display the same when using a Time series chart in Grafana.The new aggregated metric is queried like any other Prometheus metric in Promscale.PromQL:node_memfree_1hour{__column__=""avg""}SQL:SELECT time, jsonb(labels) as metric, avg
FROM node_memfree_1hour m
INNER JOIN prom_series.node_memory_MemFree s 
    ON (m.series_id=s.series_id)
ORDER BY time ascNote: typically you will want to query a certain time window, not return all the data. With PromQL the time window must be defined as part of thequery_range API call, not the query while in SQL it is specified in the query itself via aWHEREclause. For example, the SQL for querying the last 24 hours of data would beSELECT time, jsonb(labels) as metric, avg
FROM node_memfree_1hour m
INNER JOIN prom_series.node_memory_MemFree s 
    ON (m.series_id=s.series_id)
WHERE time > NOW() - INTERVAL '24 hours'
ORDER BY time ascWhen using SQL in Grafana, in order for the query to use the time window selected in the time picker the WHERE clause would beWHERE $__timeFilter(time).In PromQL, the special__column__label in the query specifies which column of the view will be returned. As we saw when defining the continuous aggregate, we can define multiple statistical aggregates in the same continuous aggregate as different columns. The__column__selector specifies which one of those columns to return. By default, the column named value is used.What this PromQL query will return is all the series with their average aggregates per hourly time buckets.{
   ""status"" : ""success"",
   ""data"" : {
      ""resultType"" : ""matrix"",
      ""result"" : [
         {
            ""metric"" : {
               ""__name__"" : ""cpu_usage_1hour"",
               ""__schema__"" : ""public"",
               ""__column__"" : ""avg"",
               ""node"" : ""prometheus"",
               ""instance"" : ""xyz""
            },
            ""values"" : [
               [ 1628146800, ""15.98"" ],
               [ 1628150400, ""16.02"" ],
               [ 1628154000, ""16.05"" ]
            ]
         },
         ...The results return the hourly average over the node_memfree metric. As we can see, the results contain some special labels. The__schema__label gives us the schema of the metric view and the__column__label gives the column of the continuous aggregate which we are querying.Note that, thanks toreal-time aggregates, the results will return the latest data as well, even if it has not yet been materialized.Taking advantage of TimescaleDB hyperfunctionsTimescaleDB hyperfunctions, a series of SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in PostgreSQL with fewer lines of code, provide several advanced aggregates that may be of special interest to Promscale users. For example, there is ahyperfunction to calculate approximate percentiles over time. We may want to see the 1st percentile of free memory to find nodes that underutilize memory (if the 1st percentile is high, that means the machine has a lot of free memory most of the time). To do this, we could get the 1st percentile of free memory by defining an aggregate like:CREATE MATERIALIZED VIEW node_memfree_30m_aggregate
WITH (timescaledb.continuous)
AS SELECT
    time_bucket('30 min', time) as bucket,
    series_id,
    percentile_agg(value) as pct_agg
FROM prom_data.node_memory_MemFree
GROUP BY time_bucket('30 min', time), series_id;The aggregate contains a sketch that can be used to answer the percentile query for any percentile.  For example, to create a view showing the first and fifth percentile:CREATE VIEW node_memfree_30m AS 
SELECT
    bucket + '30 min' as time,
    series_id,
    approx_percentile(0.01, pct_agg ) as first,
    approx_percentile(0.05, pct_agg ) as fifth 
FROM node_memfree_30m_aggregate;

SELECT register_metric_view('public', 'node_memfree_30m');We can now perform queries such asPromQL:node_memfree_30m{__column__=""first""}SQL:SELECT time, jsonb(labels) as metric, first
FROM node_memfree_30m m
INNER JOIN prom_series.node_memory_MemFree s 
    ON (m.series_id=s.series_id)
ORDER BY time ascYou might ask why add the complexity of creating two views for the same aggregate? The answer is that this allows us to change the accessors we expose to PromQL without having to recalculate the materialization. For example, we could add a median (50% percentile) column by changing thenode_memfee_30mview without changing thenode_memfree_30m_aggregatematerialization. In addition, we can derive more coarse-grained aggregations using fine-grained materializations: for example, we can create a 1-hour view based on the 30-minute materialization and have the results be accurate.For more information about two-step aggregation, and why we use it, seeour blog post on the topic.Aggregating countersTimescaleDB hyperfunctions also contain acounter aggregate hyperfunctionthat is able to store information for Prometheus-style resetting counters. This aggregate is able to derive all the counter-specific Prometheus functions: rate, irate, increase and resets:CREATE MATERIALIZED VIEW cpu_usage_30m_aggregate
WITH (timescaledb.continuous)
AS SELECT
    time_bucket('30 min', time) as bucket,
    series_id,
    counter_agg(time, value, time_bucket_range('30 min', time)) as cnt_agg
FROM prom_data.cpu_usage
GROUP BY time_bucket('30 min', time), series_id;CREATE OR REPLACE VIEW cpu_usage_30m AS
SELECT
    bucket + '30 min' as time,
    series_id,
    extrapolated_delta(cnt_agg, method =>'prometheus') as increase,
    extrapolated_rate(cnt_agg, method => 'prometheus') as rate,
    irate_right(cnt_agg) as irate,
    num_resets(cnt_agg)::float as resets
FROM cpu_usage_30m_aggregate;

SELECT prom_api.register_metric_view('public', 'cpu_usage_30m');We can now perform queries such asPromQL:cpu_usage_30m{__column__=""rate""}SQL:SELECT time, jsonb(labels) as metric, rate
FROM cpu_usage_30m m
INNER JOIN prom_series.cpu_usage s ON (m.series_id=s.series_id)
ORDER BY time asc;To make it easier to create and manage continuous aggregates, we plan to create more simplified interfaces in the next few releases and are actively looking for feedback from the community in thisGithub discussion.Data retention for downsampled dataWe now have a new metric which is derived from existing raw metric data points. By default, it will use the same data retention policy as our other metrics, which might not be what we want. Usually, downsampled data is kept for longer since it allows for long term analysis without incurring the storage and performance costs of raw data. Therefore, we want to keep access to the aggregates of the data even after our raw metric data has been dropped.To enable this, all we need to do is increase theretention periodfor our new metric. We do this by setting a retention period like we would do on any other metric in the system.SELECT set_metric_retention_period('public', 'cpu_usage_1hour', INTERVAL '365 days');This will increase the retention period of our continuous aggregate to a full year, even if the underlying metric data on which it was based has been deleted after it reached its retention period.ConsiderationsBefore using Promscale continuous aggregates there are a few considerations to take into account.First, if the__column__label matcher is not specified, it will default tovalue, which means it will try to query the column namedvalue. If the column does not exist, we will get an empty result (since it won't match anything in the system). To take advantage of this fact, consider creating continuous aggregates with avaluecolumn set to the value you want as the default value when matching the metric. In ournode_memfree_1hourexample, we could have used the following continuous aggregate instead:CREATE MATERIALIZED VIEW node_memfree_1hour
WITH (timescaledb.continuous) AS
  SELECT 
        timezone('UTC', 
          time_bucket('1 hour', time) AT TIME ZONE 'UTC' +'1 hour')  
            as time , 
        series_id,
        min(value) as min, 
        max(value) as max, 
        avg(value) as value
    FROM prom_data.node_memory_MemFree
    GROUP BY time_bucket('1 hour', time), series_idWith this configuration, getting the value of the average ofnode_memfree_1hourwith PromQL would simply benode_memfree_1hourSo no need to pass a__column__selector.Second, both the__schema__and__column__label matchers support only exact matching, no regex or other multi value matchers allowed. Also, metric views are excluded from queries that match multiple metrics (i.e., matching on metric names with a regex).{__name__=~""node_mem*""} // this valid PromQL query will not match our previously created metric viewFinally, if we ingest a new metric with the same name as a registered metric view, it will result in the creation of a new metric with the same name but a different schema (all ingested metrics are automatically added to theprom_dataschema, while the new aggregated metric would be in the public schema by default). This would likely cause confusion when querying a metric by its name since by default Promscale will query the metric in theprom_dataschema (we could specify a different__schema__label in our query). To avoid it, make sure you name your continuous aggregate views with a name that raw ingested metrics will not have, likenode_memfree_1hour.ConclusionIn this blog post, we have looked at two ways to downsample Prometheus metrics: Prometheus recording rules and Promscale continuous aggregates which offer additional capabilities.Thankfully, Promscale supports downsampling and custom retention policies with both recording rules and continuous aggregates so you can choose the right solution for your needs.There are a few things to take into account when deciding on a downsampling solution:Access to recent data.If this materialization will be used in operational or real-time dashboards prefer continuous aggregates because of thereal-time aggregatefeature.Size of the time-bucket. Continuous aggregates materialize the intermediate, not the final form, so querying the data is a bit more expensive than with recording rules. Thus, continuous aggregates are better when aggregating more data points together (1 hour or more of data), while recording rules are better for small buckets.Number of metrics in materialization. Currently, continuous aggregates can only be defined on a single metric. If you need to materialize queries on more than one metric, use recording rules. However, you should also consider whether joining the materialized metrics (the result of the materialization instead of the raw input) may be a better approach.Query flexibility.If you know the exact queries that will be run on the materialization, recording rules may be more efficient. However, if you want flexibility, continuous aggregates can answer more queries based on the materialized data.Access to old data.If you need old data points to also be aggregated as soon as downsampling for a metric is configured, continuous aggregates would be a better choice, especially if this is something you think you will be doing often, since recording rules require additional steps to backfill data.Promscale continuous aggregates are currently in beta and we are actively looking for feedback from the community to help us make it better. Share your feedback in thisGitHub discussion.Get started with PromscaleIf you’re new to Promscale and want to get started with our new multi-tenancy functionality today:Install Promscale today via Helm Charts, Docker, and others.Follow the instructions in ourGitHub repository. As a reminder, Promscale is open-source and completely free to use. (GitHub ⭐️  welcome and appreciated! 🙏.)If you are looking for managed Prometheus storage, get started now withPromscale on Timescale (free 30-day trial, no credit card required). Up to 94 % cost savings on managed Prometheus with Promscale on Timescale.See our docsto enable multi-tenancy support on your Promscale instance today.Check out ourGetting Started with Promscale tutorialfor more on how Promscale works with Prometheus, installation instructions, sample PromQL and SQL queries, and more.WatchPromscale 101 YouTube playlistfor step-by-step demos and best practices.Whether you’re new to Promscale or an existing community member, we’d love to hear from you!Join TimescaleDB Slack, where you’ll find 7K+ developers and Timescale engineers active in all channels. (The dedicated #promscale channel has 2.5K+ members, so it’s a great place to connect with like-minded community members, ask questions, get advice, and more).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/a-different-and-often-better-way-to-downsample-your-prometheus-metrics/
2022-06-08T13:05:18.000Z,How Prometheus Querying Works (and Why You Should Care),"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.🍿 This post is an adaptation of a talk presented by Harkishen Singh at Prometheus Day Europe 2022.Click here to watch the recording.Have you ever wondered why a simple PromQL query (like the one below) takes more time to execute the more instances you monitor, even if the number of instances is not included in the query? And why does the performance slow down every time you run the query with an increased time range?node_cpu_seconds{job=”prom”}The answer has to do with one thing: how Prometheus queries data.Getting familiar with the Prometheus query execution flow will help you understand your PromQL queries better. After reading this post, you will know more about how Prometheus stores data, its indexing strategies, and why your PromQL queries perform as they do. You will also get tips on what to look for when optimizing your query performance.Prometheus Storage: The Data ModelLet’s start by describing the basics of thePrometheus data model. Prometheus uses a simple model based on labels, values, and timestamps—the labels collectively form a series with multiple samples. Each sample is a pair of timestamp and value. If a label name or value changes in a previously defined series, Prometheus will consider it an entirely new series.Series, labels, values, and timestamps in PrometheusPrometheus stores this data in itsinternal time-series database(although it also supports external storage systems likePromscale).Prometheus' database is partitioned into basic units of storage calledblocksthat contain the data corresponding to a particular time range (for example, two hours). Blocks store two important elements:Sample data:the actual metric values and associated timestamps, monitored from targets.Indexes:the mechanism through which Prometheus can access the data.Blocks also contain ameta.jsonfile (with metadata about the block) andtombstones(which have deleted series alongside information about them).Inside blocks, the sample data is stored in a secondary unit of partitioning called chunks. A chunk is a collection of samples in a time rangefor a particular series. A chunk will contain data from one series only.The Prometheus internal database uses basic units of storage called blocks—and within blocks, the sample data is further partitioned into chunksPrometheus Storage: Indexing StrategiesLet’s now look at the index file more closely.Inside this index file, Prometheus stores two types of indexes:The postings indexrepresents the relationship between labels and the set of series that contain that label. For example, the label__name__=""up""could be contained in two series,{__name__=""up"",job=”prom”, instance=”:9090”}and{__name__=”up”,job=”prom”, instance=”:9090”}.The series indexindicates which chunks belong to a particular series. For example, the series{__name__=”up”,job=”prom”,instance=”:9090”}could be stored in two chunks{chunk-1.1, chunk-1.2}. Remember that none of these chunks are shared between two series; each chunk is specific to one series only.Prometheus Queries: ExecutionSo, how is the data retrieved when we run a query in Prometheus?Prometheus query execution flow involves four basic steps:Identifying which blocks correspond to the query’s time range.Using the postings index to find the series that match the labels.Using the series index to identify which chunks correspond to each series.Retrieving the sample data from the chunks.Query execution flow in PrometheusLet’s break it down using a query example:up{job=”prom”, instance=”:9090”}
start=00:00, end=00:30.Step 1: Identifying which blocks correspond to the query’s time rangePromQL queries contain labels( __name__=”up”,job=”prom”, andinstance=”:9090”)and a time range(start=00:00, end=00:30).To execute the query, Prometheus will first identify which blocks should be evaluated to satisfy the query’s time range.Step 2: Using the postings index to find the series that match the labelsNext, each block (from step one) will be scanned to find the series that matches the labels. The postings index, located within the index file, contains this information.In our example query, we have three labels:__name__=”up”job=”prom”instance=”:9090”This query will be given to the index reader, which will scan the postings index for each of the blocks identified in step one and look for a match between the query’s labels and those inside the postings table.Suppose that three series match the first label for a particular block:series 1,series 2,series 3.  For the second label, there are two series that match:series 2,series 3. And for the third label, there is only one match:series 3. Since the PromQL expression implies an AND condition, Prometheus will be looking for the series that contains the three labels: in this case,series 3.Imagine the following for a particular block:For the first label:series 1,series 2,series 3matchFor the second label:series 2,series 3matchFor the third label:series 3matchesIf we see the PromQL query, it needs a series that has the first, second, and third labels. So, there is an AND condition. Hence, we take the results intersection of the three labels. We getseries 3as the output.The process will repeat for each block identified in step one.Step 3: Using the series index to identify which chunks correspond to each seriesPrometheus now has a series set that satisfies the query’s label for each block. The next step is to identify the chunks that correspond to each series. To get that information, Prometheus looks at the series index within the index file.Step 4. Retrieving the sample data from the chunksAfter step three, Prometheus has a set of chunks that store the data we’re looking for. The data is returned, and the query is completed.Dependencies between each of the steps involved in the query execution flowThe process starts with blocks, continues with labels and series, and finishes with chunks. This means that a delay in one of the steps will have a multiplying effect on the following steps, which helps us answer the questions at the start of this blog post.Why does the performance of a query like{job=”prom”}slow down every time we run the query with an increased time range?Hopefully, you now understand that a query with a long time range implies that there will be more blocks to consider in evaluating a particular query. This will cause performance to slow down, as Prometheus will have to go through each one of the index files within each block, reading the postings indexes, the series indexes, and finally retrieving a set of chunks.Something similar happens if you have high cardinality. This relates to our second question:Why does{job=”prom”}take more time to execute the more instances you monitor, even if the number of instances is not included in the query?We mentioned how every new value attached to a label creates a whole new series in Prometheus. When a particular label has many values attached, it directly translates to a high number of series.Now that you know how the query execution process works, you can see how having high cardinality (i.e., a high number of series) slows things down. Under high cardinality, Prometheus’ index reader has to go through a huge postings table, retrieving a long list of series. For step three alone, Prometheus will need to look at each of the series to retrieve a set of chunks, which by now is surely long.Long story short, having high cardinality leads to the sort of multiplicative slow-down effect we described earlier.3 Tips to Optimize Your Prometheus QueriesBefore wrapping up, we can highlight some best practices based on our lessons:1. Avoid unnecessary label valuesMore label values for a particular label will lead to more cardinality, affecting the PromQL query performance. Hence, you should avoid storing ephemeral values in a label, such as storing logs as label values.If your target exposes ephemeral labels, you can consider dropping them using thedropaction inrelabel_configin Prometheusscrape_config.2.Keepyour scrape interval high—you can use downsampling for thisThe scrape interval is the interval in which Prometheus scrapes a metric target. If scrape intervals are low (for example, one second), it can lead to lots of chunks in the Prometheus block. Looping through chunks will be intense, and queries over a long time range can get slow.To solve this issue, you can try to downsample your data usingrecording rules. Take into account, however, that recording rules will only work on data ingested after the recording rules are created.If you are usingPromscaleto store your Prometheus metrics, you can also usecontinuous aggregatesto speed up your queries. Continuous aggregations in Promscale allow you to downsample Prometheus data independently of the ingestion time and query them using PromQL and SQL.Check out this blog postto read more about the advantages of using continuous aggregates versus recording rules for downsampling.3.If you are using Promscale, try modifying the TimescaleDB chunk sizePromQL evaluation slows down if more blocks need to be read. As we saw earlier, Prometheus’ index reader has to load the index files of each block in the memory to process it when dealing with a higher number of blocks, which can get expensive.To solve this issue, it would be beneficial to alter the block size, tailoring it to the needs of each workload—for example, making it bigger if we end up in the situation described above. Unfortunately, users cannot modify the block size in the Prometheus time-series database, but if you are using Promscale, this is different.If you are storing your Prometheus metrics in Promscale,you will be using TimescaleDB as the underlying database. And in TimescaleDB, you have complete control over your indexing and yourchunk size(chunksare the time-based partitioning unit in TimescaleDB). For example, you could experiment with your chunk size; often, smaller chunk sizes can help with memory usage, taking less shared buffer memory.💡 Even if the concepts of Prometheus’ blocks and TimescaleDB’s chunks are similar, the indexing strategy that TimescaleDB uses when scanning chunks is fundamentally different from that of Prometheus. Having many blocks may be detrimental to query performance in Prometheus, but having a high number of chunks is not necessarily detrimental to query performance in TimescaleDB.To learn more about how chunks work in TimescaleDB, check out our docs.It's a Wrap!By understanding how Prometheus indexes work, you now have an intuitive understanding of how your PromQL queries will perform, making it easier to keep an eye on the parameters that may affect your performance.Do you need an external data store for Prometheus? Check outPromscale, the unified observability backend for Prometheus metrics and OpenTelemetry traces built on PostgreSQL and TimescaleDB. Promscale has all the features you need including 100 % PromQL compliance, Grafana integration, PromQL alerts and recording rules, exemplars, Prometheus high availability and multi-tenancy, and many more.Get started now withPromscale on Timescale (free 30-day trial, no credit card required)orself-hostfor free. Up to 94% cost savings on managed Prometheus with Promscale on Timescale.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-prometheus-querying-works-and-why-you-should-care/
2022-02-24T14:03:54.000Z,Learn OpenTelemetry Tracing With This Lightweight Microservices Demo,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.OpenTelemetry is an open source observability framework for cloud-native service and infrastructure instrumentation hosted by theCloud Native Computing Foundation(CNCF). It has gained a lot of momentum with contributions from all major cloud providers (AWS, Google, Microsoft) as well as observability vendors (including Timescale) to the point it has become the second project with themost activityand contributors only after Kubernetes.Last October, we addedbeta support for OpenTelemetry tracesto Promscale, the observability backend powered by SQL. A trace (or distributed trace) is a connected representation of the sequence of operations that were performed across all microservices involved in order to fulfill an individual request. Each of those operations is represented using a span. Each span includes a reference to the parent span except the first one which is called the root span. As a result, a trace is a tree of spans. We told you everything about OpenTelemetry traces inthis blog post.However, it’s better to learn by doing. When talking about OpenTelemetry traces and Promscale, we often wished we had access to a microservices demo application instrumented with OpenTelemetry, so users could play with it to directly experience the potential of tracing. For example,in the blog post we referenced earlier, we usedHoneycomb’s fork of Google's microservices demo, which adds OpenTelemetry instrumentation to those services. But while it is a great demo of a microservices environment, we found it too complex and resource-heavy to run it locally on a laptop for the purposes of playing with OpenTelemetry, since it requires a Kubernetes cluster with 4 cores and 4 GB of available memory.This inspired us to build a more accessible and lightweight OpenTelemetry demo application:https://github.com/timescale/opentelemetry-demo/In this blog post, we’ll introduce you to this demo, which consists of a password generator overdesigned as a microservices application. We’ll dive into its architecture, we’ll explain how we instrumented the code to produce OpenTelemetry traces, how to get the demo up and running on your computer in a few minutes, and how to analyze those traces with Jaeger, Grafana, Promscale, and SQL to understand how the system behaves and to troubleshoot problems.But that's not all: later in the post, we will walk you through the six (!) prebuilt Grafana dashboardsthat come with the demo. These are powerful dashboards that will give you a deep understanding of your systems by monitoring throughput, latency, and error rate. You will also be able to understand upstream and downstream service dependencies.Even if we built this demo to make it easier for users to experiment with Promscale, this demo application is not Promscale-specific. It can be easily configured to send the telemetry it generates to any OpenTelemetry-compatible backend and we hope the broader community, even those not using Promcale, will find it useful.To learn about Promscale, check out ourwebsite. If you start using us,join the #promscale channel in our Slack Communityand chat with us about observability, Promscale, and anything in between - the Observability team is very active in #promscale! Plus, we just launched ourTimescale Community Forum. Feel free to shoot us any technical questions there as well.One more thing: if you share our mission of serving developers worldwide 🌏 and want to join our fully remote, global team...We are hiring broadly across many roles!Demo ArchitectureThedemo applicationis a password generator that has been overdesigned to run as a microservices application. It includes five microservices:Thegenerator serviceis the entry point for requests to generate a new password. It calls all other services to create a random password.Theupper servicereturns random uppercase letters (A-Z).Thelower servicereturns random lowercase letters (a-z).Thedigit servicereturns random digits (0-9).Thespecial servicereturns random special characters.Apart from the microservices, the demo also deploys a pre-configured OpenTelemetry observability stack composed of the OpenTelemetry Collector, Promscale, Jaeger, and Grafana:Architecture diagram of the demo environmentObviously, this is a silly design - this is not how you would design a password generator. But we decided to use this application because it’s a very easy-to-understand example (everybody is familiar with creating secure passwords), the code is simple, and it’s very lightweight, so you can easily run it on a computer with Docker (no Kubernetes required).To make the traces generated more interesting, the code of the services introduces random wait times and errors. There is also a bug in the code of one of the services (an Easter egg  👀 for you to find!).All services are built in Python, except the lower service, which is built in Ruby. The demo also includes a load generator that makes requests to the generator service to create new passwords. It instantiates three instances of that load generator.InstrumentationThe first step to start getting visibility into the performance and behaviors of the different microservices is to instrument the code with OpenTelemetry to generate traces.  The five microservices included in the demo have already been instrumented; in this section, however, we'll explain how we did it, in case you’re interested in learning how to instrument your own.OpenTelemetry provides SDKs and libraries to instrument code in a variety of different languages. SDKs allow you to manually instrument your services; at the time of this writing, the OpenTelemetry project provides SDKs for 12 (!) languages (C++, .NET, Erlang/Elixir, Go, Java, Javascript, PHP, Python, Ruby, Rust, and Swift). The maturity of those SDKs varies, though—check theOpenTelemetry instrumentation documentationfor more details.Additionally, OpenTelemetry provides automatic instrumentation for many libraries in a number of languages likePython,Ruby,Java,Javascript, or .NET. Automatic instrumentation typically works through library hooks or monkey-patching library code. This helps you get a lot of visibility into your code with very little work, dramatically reducing the amount of effort required to start using distributed tracing to improve your applications.For services written in Python, you could leverage auto-instrumentation and not have to touch a single line of code. For example, for the generator service and after setting theOTEL_EXPORTER_OTLP_ENDPOINTenvironment variable to the URL of the OTLP endpoint (the OpenTelemetry Collector for example) where you want data to be sent, you would run the following command to auto-instrument the service:opentelemetry-instrument --traces_exporter otlp python generator.pyAs we just mentioned, we already instrumented the code of the different microservices in our password generator demo. In our case, we decided to manually instrument the code, to show how it is done and in order to add some additional instrumentation, but that is not required.To instrument the Python services, first we imported a number of OpenTelemetry libraries:from opentelemetry import trace
from opentelemetry.trace import StatusCode, Status
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporterThen, we initialized and set up the tracing instrumentation for the service, including setting up the Flask and HTTP auto-instrumentation:trace.set_tracer_provider(TracerProvider(resource=Resource.create({""service.name"": ""generator""})))
span_exporter = OTLPSpanExporter(endpoint=""collector:4317"")
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(span_exporter))
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
tracer = trace.get_tracer(__name__)And finally, we added some manual spans withtracer.start_as_current_span(name)for each operation, and some events withspan.add_event(name):def uppers() -> Iterable[str]:
    with tracer.start_as_current_span(""generator.uppers"") as span:
        x = []
        for i in range(random.randint(0, 3)):
            span.add_event(f""iteration_{i}"", {'iteration': i})
            try:
                response = requests.get(""http://upper:5000/"")
                c = response.json()['char']
            except Exception as e:
                e = Exception(f""FAILED to fetch a upper char"")
                span.record_exception(e)
                span.set_status(Status(StatusCode.ERROR, str(e)))
                raise e
            x.append(c)
        return xCheck the code of thegenerator,upper,digitorspecialservices for complete examples.For the Ruby service, we did something similar. Checkthe code of the lower servicefor more details!Running the demoThe demo usesdocker-composeto get all components up and running on your computer. Therefore, before running it, you first need to install theDocker EngineandDocker Composeas prerequisites. We’ve tested the demo on recent versions of MacOS (both Intel and M1 processors), Linux (tested on Ubuntu) and Windows.Theopentelemetry-demo GitHub repohas everything you need to run the demo. First, clone the repo:git clone https://github.com/timescale/opentelemetry-demo.gitOr if you prefer, justdownload itand unzip it.Next, go into the opentelemetry-demo folder (or opentelemetry-demo-main if you used the download option):cd opentelemetry-demoAnd run:docker-compose up --detachNote: on Linux systems the default installation only gives the root user permissions to connect to the Docker Engine and the previous command would throw permission errors. Running sudo docker-compose up --detach would fix the permissions problems by running the command as root but you may want togrant your user account permission to manage Docker.This will execute the instructions in thedocker-compose.yamlfile, which will build a Docker image for each service. This could take a few minutes. Then, run all the different components of the demo environment. Whendocker-composecompletes, you should see something like the following:Once the microservices demo installation is complete, you will see this output from docker-compose.And that’s it. Congratulations! You have a microservices application instrumented with OpenTelemetry sending traces to an OpenTelemetry observability stack.Installing the microservices demo with docker-compose.If at any point you want to stop the demo, just rundocker-compose down.We'll talk about Jaeger and Grafana extensively in the next sections of this blog post, but as a quick note—Jaeger runs onhttp://localhost:16686/and Grafana onhttp://localhost:3000/. Grafana will require credentials to log in for the first time: useadminfor the username and alsoadminfor the password.  You will be able to update your password immediately after.Visualize individual traces with JaegerNow that you have the entire demo up, including the password generator microservices and the preconfigured OpenTelemetry observability stack, you are ready to start playing around with the generated traces.As we explained earlier, traces represent a sequence of operations, typically across multiple services, that are executed in order to fulfill a request. Each operation corresponds to a span in the trace. Traces follow a tree-like structure, and thus most tools for visualizing traces offer a tree view of all the spans that make up a trace.Jaeger is the most well-known open-source tool for visualizing individual traces. Actually, Jaeger is more than just visualization, as it also provides libraries to instrument code with traces (whichwill be deprecated soon in favor of OpenTelemetry), a backend service to ingest those traces, and an in-memory and local disk storage. Jaeger also provides a plugin mechanism and a gRPC API for integrating with external storage systems. (Promscale implements such gRPC API to integrate with Jaeger; actually, the Promscale teamcontributed it to the project!)Jaeger is particularly helpful when you know your system is having a problem and you have enough information to narrow down the search to a specific set of requests. For example, if you know that some users of your application are experiencing very slow performance when making requests to a specific API, you could run a search for requests to that API that have taken more than X seconds, and from there, visualize several individual traces to find a pattern of which service and operation is the bottleneck.Let’s turn to our simple demo application to show an example of this. If you have the demo application running,you can access Jaeger here.Now, let’s say we know there are some requests to generate a password that are taking more than 10 seconds. To find the problematic requests, we couldfilter traces in Jaeger to the generator service that have taken more than 10 seconds, and sort them by “Longest First”:Jaeger search UI (if you have the microservices demo running,you can access Jaeger here).In our case, we are seeing a request with 200 spans taking more than 30 seconds. Let’s open that trace and see why it is taking so long:Visualizing a distributed trace in Jaeger (if you have the microservices demo running,you can access Jaeger here).After inspecting it carefully, we can see that therandom_digitoperation in the digit service is taking more than 1 second every time it runs, while other operations are usually run in a few microseconds or milliseconds. If we go back to the list of traces, and open another slow trace, we will see that it points again to the same problem.With that information, we can now go inspect the code of therandom_digitoperation in the digit service, which reveals the problem:def random_digit() -> str:
    with tracer.start_as_current_span(""random_digit"") as span:
        work(0.0003, 0.0001)

        # slowness varies with the minute of the hour
        time.sleep(sin(time.localtime().tm_min) + 1.0)

        c = random.choice(string.digits)
        span.set_attribute('char', c)
        return cNow we can see why this operation is giving us trouble: it's because the code is calling thetime.sleepfunction, which is slowing it down!Besides trace visualization, Jaeger also providesa service mapof your microservices architecture generated from trace data. This is useful to get an overview of service dependencies, which can help identify unexpected dependencies and also ways to simplify or optimize the architecture of your application. For example, the map below shows us how the generator service calls all the other services:Service map view in Jaeger (if you have the microservices demo running,you can access Jaeger here).Analyze the performance and behavior of the application with Grafana and SQLJaeger is a fantastic tool to understand individual traces, but it doesn’t provide a query language to analyze and aggregate data as needed. Traces have a wealth of information that you can use to better understand how your system is performing and behaving -  if you have the ability to run arbitrary queries on the data. For example, you can’t compute the throughput, latency, and error rate of your services with Jaeger.This is where Promscale can help you. The observability stack deployed by the password generator demo stores all OpenTelemetry traces in Promscale through its endpoint that implements theOpenTelemetry Protocol(OTLP). And because Promscale is built on top of TimescaleDB and PostgreSQL, it allows you to run full SQL to analyze the data.In this section, we’ll tell you how to use Grafana to analyze traces with a set of predefined dashboards that query the data in Promscale using SQL. These dashboards allow you to monitor the three golden signals often used to measure the performance of an application: throughput, latency, and error rate. We've also built dashboards that will help you understand your upstream and downstream service dependencies.We have built six predefined dashboards to do different analysis,all of them included in our microservices demo:Request rates or throughputRequest durations or latencyError ratesService dependenciesUpstream dependenciesDownstream dependenciesLet's get into them.Request ratesRequest rates dashboard in Grafana.You can access it hereif you have the microservices demo running.It’s possible to compute throughput of your services from traces, if you can query and aggregate them like you can do with SQL. Before showing you how to do that though, it is important to notice that this requires that all trace data is stored, or the results will not be accurate. It is not uncommon to use downsampling to only keep a representative set of traces to reduce the compute and storage requirements, but if traces are downsampled, the computed request rate won’t be exact.In our demo, theRequest Rate dashboardshows the evolution of throughput (i.e. request rate) over time. It has two charts: the first one computes the requests per second using one-minute time buckets (lower resolution, equivalent to the average requests per second in each minute), and the other one uses one-second time buckets.To generate the requests per second with one-second time buckets (the bottom chart), we used the following SQL query:SELECT
    time_bucket('1 second, start_time) as time,
    count(*) as req_per_sec
FROM ps_trace.span s
WHERE $__timeFilter(start_time)
AND parent_span_id is null -- just the root spans
GROUP BY 1
ORDER BY 1The query follows the well-knownSELECT,FROM,WHERESQL structure. Promscale provides theps_trace.spanview that gives you access to all spans stored in the database. If you are not familiar with SQL views, from a query perspective, you can think of them as relational tables for simplicity.This query is just counting all spans(count(*))in one-second buckets(time_bucket('1 second, start_time))in the selected time window($__timeFilter(start_time))that are root spans(parent_span_id is null). Root spans are spans that don’t have a parent: they are the very first span of a trace. So effectively, this condition filters out internal requests between the microservices that make up the password generator while keeping those that originate from outside of the system - that is, the actual requests that users make to the application.If you want to run the query outside of Grafana, replace$__timeFilter(start_time)with the corresponding condition onstart_time. For example, to limit the query to the last 30 minutes, replace$__timeFilter(start_time)withstart_time < NOW() - INTERVAL '30 minutes'.Request durationsRequest durations dashboard in Grafana.You can access it hereif you have the microservices demo running.Latency is also key to understanding if your application is responding fast and delivering a good experience to your users. The most common way to look at latency is by using histograms and percentiles. Histograms indicate the distribution, while percentiles help you understand what percentage of requests took X amount of time or less. Common percentiles are 99th, 95th and 50th, which is usually known as the median.TheRequest Durations dashboarduses a number of different Grafana panels to analyze request latency in our password generator demo.How did we build this? The easiest way to display a histogram is to use Grafana’s Histogram panel with a query that returns the duration of each request to the system in the selected time window:SELECT duration_ms
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND parent_span_id is nullThe histogram view shows that the majority of user requests take less than one second. However, there is a long tail of requests that are taking a long time, up to 30 seconds, which is a very poor user experience for a password generator:Latency histogram in Grafana (from theRequest Durationsdashboard).The previous query is not very efficient if you have lots of requests. A more optimized way of doing this would be to use TimescaleDB’shistogramfunction and compute the histogram buckets in the database:SELECT histogram(duration_ms, 100, 10000, 20)
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND parent_span_id is nullThe query above would generate a histogram for the duration_ms column with a lower bound of 100 ms, an upper bound of 10,000 ms (i.e., 10 seconds), and 20 buckets. The output of that query would look something like this:{7,98,202,90,33,23,21,20,15,16,9,10,11,11,8,5,5,4,5,6,4,91}Each of those values corresponds to one bucket in the histogram. Unfortunately, there isn’t an out-of-the-box Grafana panel that can use this input to display a chart. However, with some SQL magic it can be converted into something that could easily be displayed in a Grafana panel.When using metric instrumentation instead of trace instrumentation, you have to define the buckets when you initially define the metric and therefore you can’t change the histogram buckets later for data that has already been stored. For example, imagine that you set a lower bound of 100ms, but you realize later when investigating an issue that it would have been valuable to you to have a lower bound of 10ms.With traces though, you can change the structure of your histogram as needed, since they are computed at query time. Additionally, you have the flexibility to filter the histogram to show the latency of requests having certain tags - like the latency of requests made by a specific customer, for instance.TimescaleDB also providesfunctions to compute percentiles. The following query can be used to see the evolution of different percentiles over time:SELECT
    time_bucket('1 minute', start_time) as time,
    ROUND(approx_percentile(0.99, percentile_agg(duration_ms))::numeric, 3) as duration_p99,
    ROUND(approx_percentile(0.95, percentile_agg(duration_ms))::numeric, 3) as duration_p95,
    ROUND(approx_percentile(0.90, percentile_agg(duration_ms))::numeric, 3) as duration_p90,
    ROUND(approx_percentile(0.50, percentile_agg(duration_ms))::numeric, 3) as duration_p50
FROM span
WHERE
    $__timeFilter(start_time)
    AND parent_span_id is null
GROUP BY time
ORDER BY timeLatency percentiles in Grafana (from theRequest Durationsdashboard).Finally, you can also use SQL to just return the slowest requests across all traces, something that is actually not possible to do with Jaeger when the volume of traces is high. Jaeger limits the amount of traces you can retrieve to 1,500 and sorting is implemented in the UI. Therefore, if there are more than 1,500 matching traces in the selected timeframe, you’ll have no guarantee that you’ll actually see the slowest traces.With Promscale and SQL, you are sure to get the latest traces regardless of how many there are because sorting is performed in the database:SELECT
  trace_id,
  start_time,
  duration_ms
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND parent_span_id is null
ORDER BY duration_ms DESC
LIMIT 10;One more interesting thing. In theRequest Durations dashboard, the query we use is slightly different:SELECT
  replace(trace_id::text, '-'::text, ''::text) as trace_id,
  start_time,
  duration_ms
FROM ps_trace.span
WHERE $__timeFilter(start_time)
AND parent_span_id is null
ORDER BY duration_ms DESC
LIMIT 10;The difference is that we remove “-” from the trace id. We do that because we store the original trace id as reported by OpenTelemetry, but in the dashboard, we do a nice trick to link the trace id directly to the Grafana UI, so we can visualize the corresponding trace (Grafana expects a trace id with “-” removed).List of slowest traces (from theRequest Durationsdashboard).If you want to see this in action, openthe dashboardand click on a trace id. It’s pretty cool, and it works thanks to the high number of options for customizing your dashboard that Grafana provides. A shootout to the Grafana team!Error ratesError rates dashboard in Grafana.You can access it hereif you have the microservices demo running.Error rate is the third golden signal for measuring application and service performance. When an operation in a request does not complete successfully, OpenTelemetry marks the corresponding span(s) with an error status. In Promscale, a span with error is indicated withstatus_code = 'error'To compute the error rate per operation in the selected time window in Grafana, you would run the following query:SELECT
    x.service_name,
    x.span_name,
    x.num_err::numeric / x.num_total as err_rate
FROM
(
    SELECT
        service_name,
        span_name,
        count(*) filter (where status_code = 'error') as num_err,
        count(*) as num_total
    FROM ps_trace.span
    WHERE $__timeFilter(start_time)
    GROUP BY 1, 2
) x
ORDER BY err_rate descError rates by service operation (from theError Ratesdashboard).All OpenTelemetry spans include aservice_nameto indicate the name of the service that performed the operation represented by the span, and aspan_nameto identify the specific operation.Service dependenciesService dependencies dashboard in Grafana.You can access it hereif you have the microservices demo running.So far we have learned how to analyze data across all services, per service, or per operation to compute the three golden signals: throughput, latency, and error rate. In the next three sections, we are going to leverage the information in traces to learn about how services communicate with each other to understand the behavior of the system.We saw that Jaeger can display a service map out of the traces being ingested. Actually, with SQL and Grafana you can get a lot more valuable insights from the same trace data. Using theGrafana node graphpanel we can generate a service map similar to the one Jaeger provides that also indicates the number of requests between services:Service map using the Grafana node graph panel (from theService Dependenciesdashboard).This dependency map already tells us that something seems to be wrong, as we see that the lower service is calling the digit service - that doesn’t make any sense! If we'd do a careful inspection of the code, it would reveal that there is a bugcalling the digit serviceevery time it receives a request to generate a lowercase character for no reason. Problem solved!The node graph panel requires two inputs to build this visualization: a first query that returns the list of nodes, and a second query that returns the edges that connect those nodes.The nodes in our service map are the services that are part of the application. The query below retrieves all the differentservice_namesthat appear in spans during the selected timeframe. The node graph panel requires an id and a title for each node, and we use the same value for both:SELECT 
   service_name as id,
   service_name as title
FROM ps_trace.span
WHERE $__timeFilter(start_time)
GROUP BY service_nameThen, we need a query to get the list of edges. The edges indicate dependencies between services, that is, that service A calls service B.SELECT
    p.service_name || '->' || k.service_name || ':' || k.span_name as id,
    p.service_name as source,
    k.service_name as target,
    k.span_name as ""mainStat"",
    count(*) as ""secondaryStat""
FROM ps_trace.span p
INNER JOIN ps_trace.span k
ON (p.trace_id = k.trace_id
AND p.span_id = k.parent_span_id
AND p.service_name != k.service_name)
WHERE $__timeFilter(p.start_time)
GROUP BY 1, 2, 3, 4This query joins the span view with itself because an individual span does not have any information about the parent span other than the parent span id. And so, we need to do a join on the parent span id to identify calls between different services (i.e., the span and the parent span were generated by different services):p.trace_id = k.trace_id
AND p.span_id = k.parent_span_id
AND p.service_name != k.service_nameThe node graph panel requires an id, a source node, a target node, and optionally a main statistic and a secondary statistic to display in the graph. The id generated in the query is an arbitrary string we defined that includes the service making the requests, the service receiving those requests, and which operation the parent service called.By doing that, if service A calls different operations in service B, the service map will show multiple arrows (one for each operation) instead of just one, which helps you understand the dependencies between services in more detail. This is not the case in the demo environment (as each service only calls one operation in another service), but it could be useful in other situations. And if you prefer to group all operations together, you would just need to removek.span_namefrom the id in the query.This is the power of using SQL for all of this: you have full flexibility to get the information you are interested in!One more thing: you can compute additional statistics to help you further understand how services are interacting. For example, you may be interested in seeing not only how many times a service is calling another service, but also how much total time is spent by the receiving service in those calls, and what’s the average (you could also compute the median, or p99, or anything you want really) of each of those calls.Summary table of service dependencies (from theService Dependenciesdashboard).The table above was generated with a Grafana table panel using the query below, that again joins the span view with itself:SELECT
    p.service_name as source,
    k.service_name as target,
    k.span_name,
    count(*) as calls,
    sum(k.duration_ms) as total_exec_ms,
    avg(k.duration_ms) as avg_exec_ms
FROM ps_trace.span p
INNER JOIN ps_trace.span k
ON (p.trace_id = k.trace_id
AND p.span_id = k.parent_span_id
AND p.service_name != k.service_name)
WHERE $__timeFilter(p.start_time)
GROUP BY 1, 2, 3
ORDER BY total_exec_ms DESCInvestigating upstream dependenciesUpstream spans dashboard in Grafana.You can access it hereif you have the microservices demo running.When you are troubleshooting a problem in a microservice, it’s often very helpful to understand what is calling that service. Imagine that you saw a sudden increase in the throughput of a particular operation in a microservice: you would like to understand what caused that increase in throughput—that is, what upstream service and operation started making a lot more requests to that microservice operation.This problem is more complicated than the dependency map problem, as we need to analyze all spans that represent requests to that microservice operation; then, retrieve the calling services; then, do the same for those... And repeat. Basically, we need to recursively traverse the tree of spans to extract all those dependencies.To visualize the dependencies, we will again use the node graph panel, which requires a query for the list of nodes and a query for the list edges. These are not as straightforward as in the dependency map, because we only want to display services that are in the upstream chain of a specific service and operation. We need a way to traverse the tree of spans for that service and operation backwards.Luckily, we have the SQL superpowers of Postgres: we can use arecursive queryto do this!The structure of a recursive query in PostgreSQL is the following:WITH RECURSIVE table_name AS (

     initial_query


     UNION [ALL]
     recursive_query
)
final_queryLet's break it down:initial_query: the query that returns the base result set that will be run against the recursive query.recursive_query: a query againsttable_namethat is initially run against the result set ofinitial_query. The results of that query are run against the recursive_query again, and then that result set is run against the recursive_query again and so on until the result set returned by the recursive query is empty.final_query: the results of the initial_query execution and the subsequent recursive_query execution are unioned (if using UNION instead of UNION ALL duplicate rows are discarded). That is, all results from initial_query and all results from subsequent recursive_query executions are put together into one working table and then run against the final_query to generate the query results that are finally returned by the database.Given a service and operation, this is the query that returns all upstream services and operations by using theWITH RECURSIVEPostgreSQL statement:WITH RECURSIVE x AS
(
    SELECT
        trace_id,
        span_id,
        parent_span_id,
        service_name,
        span_name
    FROM ps_trace.span
    WHERE $__timeFilter(start_time)
    AND service_name = '${service}'
    AND span_name = '${operation}'
    UNION ALL
    SELECT
        s.trace_id,
        s.span_id,
        s.parent_span_id,
        s.service_name,
        s.span_name
    FROM x
    INNER JOIN ps_trace.span s
    ON (x.trace_id = s.trace_id
    AND x.parent_span_id = s.span_id)
)
SELECT
    md5(service_name || '-' || span_name) as id,
    span_name as title,
    service_name as ""subTitle"",
    count(distinct span_id) as ""mainStat""
FROM x
GROUP BY service_name, span_nameThe conditionx.parent_span_id = s.span_idin therecursive_queryis the one that ensures only parent spans are returned.Each node represents a specific service and operation. Then for each node, this query will display the number of executions for the corresponding operation(count(distinct span_id) as “mainStat”).${service}and${operation}are Grafana variables that are defined at the top of the dashboard and that specify which is the service and operation for which we want to show upstream services and operations.Variable selectors (from theUpstream Spansdashboard).The query for edges is similar to the query for nodes:WITH RECURSIVE x AS
(
    SELECT
        trace_id,
        span_id,
        parent_span_id,
        service_name,
        span_name,
        null::text as id,
        null::text as target,
        null::text as source
    FROM ps_trace.span
    WHERE $__timeFilter(start_time)
    AND service_name = '${service}'
    AND span_name = '${operation}'
    UNION ALL
    SELECT
        s.trace_id,
        s.span_id,
        s.parent_span_id,
        s.service_name,
        s.span_name,
        md5(s.service_name || '-' || s.span_name || '-' || x.service_name || '-' || x.span_name) as id,
        md5(x.service_name || '-' || x.span_name) as target,
        md5(s.service_name || '-' || s.span_name) as source
    FROM x
    INNER JOIN ps_trace.span s
    ON (x.trace_id = s.trace_id
    AND x.parent_span_id = s.span_id)
)
SELECT DISTINCT
    x.id,
    x.target,
    x.source 
FROM x
WHERE id is not nullIn this case, we are settingid,targetandsourceto null in theinitial_querybecause these columns don’t exist in theps_trace.spanview. However, we need to define these columns to be able to do aUNIONwith therecursive_query, which does define them (the columns in theSELECTclause must be the same for both queries). In the final query, we only keep those rows where id is not null.This is the end result:Grafana node graph showing upstream services and operations (from theUpstream Spansdashboard).For example, we can see that the digit service is serving a similar number of requests from the lower service as from the generator service.Investigating downstream dependenciesDownstream spans dashboard in Grafana.You can access it hereif you have the microservices demo running.When investigating how to improve the performance of a specific service operation it is extremely useful to get a sense of all the different downstream services and operations that are involved in executing all executions of that service operation.To create a map of downstream dependencies using traces in Promscale, we'll use the same approach as for upstream services: we'll use the node graph panel with recursive SQL queries. The queries will be the same as the ones we used for upstream dependencies but replacing thex.parent_span_id = s.span_idcondition in therecursive_querywithx.span_id = s.parent_span_idto return the child spans instead of the parent span.If in the dashboard we selectService: generatorandOperation: /, then we basically get the entire map of how services interact, since that service and operation is the only entry point to the password generator application.Grafana node graph showing downstream services and operations (from theDownstream Spansdashboard).By analyzing downstream dependencies, we can also do other extremely valuable analyses, like looking at which downstream service operation consumes the majority of the execution time for the selected service and operation. This would be very helpful, for example, to decide where to start if we want to improve the performance of a specific service operation.However, that calculation can't be done directly by using the duration of each span, since it includes the duration of its child spans - and therefore spans that are higher in the hierarchy will always show as taking the longest to execute. To solve that problem, we subtract to the duration of each span the time spent in child spans.Let’s look at a specific example to illustrate the problem. The diagram below shows a trace with four spans representing different operations. The time effectively consumed byOperation Aist1 + t3 + t6, or expressed differently,  the duration of the span forOperation Aminus the duration of the span forOperation Bminus the duration of the span forOperation C. The time effectively consumed byOperation Bist2, byOperation Cist4, and byOperation Dist5. In this case,Operation Cis the one that has consumed most of the time in the trace.Diagram illustrating the time effectively consumed by each operation for a particular trace.To perform this calculation in aggregate for all traces starting with a specific service and operation, we can run the following recursive query:WITH RECURSIVE x AS
(
    SELECT
        s.trace_id,
        s.span_id,
        s.parent_span_id,
        s.service_name,
        s.span_name,
        s.duration_ms - coalesce(
        (
            SELECT sum(z.duration_ms)
            FROM ps_trace.span z
            WHERE s.trace_id = z.trace_id
            AND s.span_id = z.parent_span_id
        ), 0.0) as duration_ms
    FROM ps_trace.span s
    WHERE $__timeFilter(s.start_time)
    AND s.service_name = '${service}'
    AND s.span_name = '${operation}'
    UNION ALL
    SELECT
        s.trace_id,
        s.span_id,
        s.parent_span_id,
        s.service_name,
        s.span_name,
        s.duration_ms - coalesce(
        (
            SELECT sum(z.duration_ms)
            FROM ps_trace.span z
            WHERE s.trace_id = z.trace_id
            AND s.span_id = z.parent_span_id
        ), 0.0) as duration_ms
    FROM x
    INNER JOIN ps_trace.span s
    ON (x.trace_id = s.trace_id
    AND x.span_id = s.parent_span_id)
)
SELECT
    service_name,
    span_name,
    sum(duration_ms) as total_exec_time
FROM x
GROUP BY 1, 2
ORDER BY 3 DESCThis query is very similar to the one we used to get the list edges for the node graph of downstream dependencies, but it includes a calculation of the effective time spent in each operation by subtracting to the duration of each span the duration of all their child spans.We're doing this using another cool SQL feature:subqueries.s.duration_ms - coalesce(
(
    SELECT sum(z.duration_ms)
    FROM ps_trace.span z
    WHERE s.trace_id = z.trace_id
    AND s.span_id = z.parent_span_id
), 0.0) as duration_mscoalescewill return0.0if the subquery returns no results, that is if the span doesn’t have any children. In this case, the effective time spent in the corresponding operation is the total duration of the span.Graph showing the operations with the highest execution time (from theDownstream Spansdashboard).In the case of the/operation of the generator service, 87% of the time is spent in therandom_digitoperation of the digit service. Inspectingthe codereveals why; this issue is caused by the sleep code we already identified when looking at slow traces in Jaeger.Keep in mind that this method for computing effective time consumed by an operation only works if all downstream operations are synchronous. If your system uses asynchronous operations and those are being tracked as child spans, then the calculations would likely be incorrect, because child spans could happen outside the boundaries of their parent span.SummaryWith this lightweight OpenTelemetry demo, you can get an easy-to-deploy microservices environment with a complete OpenTelemetry observability stack running on your computer in just a few minutes.This will help you get started with distributed tracing visualization tools like Jaeger, which are easy to use and let you dig into specific traces to troubleshoot a problem if you know what you are looking for.With Promscale, Grafana, and SQL, you can go deeper by interrogating your trace data in any way you need. This will allow you to quickly derive new insights about your distributed systems, helping you surface and identify hard-to-find problems that you cannot find with other tools, or that would require a lot of manual work from you.If you want to learn more about Promscale, check ourwebsiteand ourdocumentation. If you start using us, don’t forget to join the #promscale channel in ourCommunity Slackto directly interact with the team building the product. You can also ask us any technical questions in our newCommunity Forum.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/learn-opentelemetry-tracing-with-this-lightweight-microservices-demo/
2022-05-16T13:09:53.000Z,Observability Powered by SQL: Understand Your Systems Like Never Before With OpenTelemetry Traces and PostgreSQL,"⚠️While part of this content may still be up to date, we regret to inform you that we decided tosunset Promscale in February 2023.Read our reasons why and other FAQs.Troubleshooting problems in distributed systems built on microservices and cloud-native technologies is a challenge, to say the least. With so many different components, all their interactions, and frequent changes introduced with autoscaling and multiple deployments, we can’t possibly predict everything that could go wrong.So far, we know that the first step to solving this problem is to start collecting detailed telemetry data from all our cloud-native systems in the form of traces, metrics, and logs. Unlike monoliths, troubleshooting distributed systems is a “trace first, metrics and logs second” process.By themselves, metrics and logs don’t provide the level of understanding required to identify anomalous behaviors in those systems. Traces, however, capture a connected view of how the entire system works to process requests, allowing software developers to figure out what isactuallyhappening when a problem occurs. Metrics and logs expand the information that traces provide—especially if directly correlated to traces.OpenTelemetry, a Cloud Native Computing Foundation (CNCF) standard for instrumentation, is driving a revolution in this space by unifying the collection of traces, metrics, and logs and making them universally available. Since OpenTelemetry focuses on helping solve the challenges of operating distributed systems, it is no surprise that the observability framework first targeted tracing support, which is currently more mature than the support for the other two signals.Having access to all that telemetry data—especially traces—solves only half of the problem.The other half requires tools to analyze, correlate and make sense of that data. That is what observability is about. It’s essentially an analytics problem: you want to be able to interrogate telemetry data with arbitrary questions and get results back in real time.Given that observability is an analytics problem, it is surprising that the current state of the art in observability tools has turned its back on the most common standard for data analysis broadly used across organizations: SQL.Good old SQL could bring some key advantages: it’s surprisingly powerful, with the ability to perform complex data analysis and support joins; it’s widely known, which reduces the barrier to adoption since almost every developer has used relational databases at some point in their career; it is well-structured and can support metrics, traces, logs, and other types of data (like business data) to remove silos and support correlation; and finally, visualization tools widely support it.However, implementing full SQL support from scratch on top of an observability store is a major undertaking that would likely take years of development.But what if we didn’t have to? What if we could leverage a well-known and reliable open-source SQL database to solve this problem?That’s exactly what we’ve done :)PostgreSQL isthe second most used and most loved databaseamong developers, backed by more than 25 years of development and a healthy and thriving community with hundreds of integrations available.You're probably thinking that observability data is time-series data that relational databases struggle with once you reach a particular scale. Luckily, PostgreSQL is highly flexible and allows you to extend and improve its capabilities for specific use cases.TimescaleDB builds on that flexibility to add time-series superpowers to the databaseand scale to millions of data points per second and petabytes of data.With PostgreSQL and TimescaleDB, we have a robust and scalable database for time-series data. Now, how do I add my OpenTelemetry data to start using SQL for observability?Promscale bridges that gap.Today, we are announcing the general availability of OpenTelemetry tracing support inPromscale, the observability backend for metrics and traces built on the rock-solid foundations of PostgreSQL and TimescaleDB. We’re now closer to becoming the unified data store for observability powered by SQL, bringing the PostgreSQL and observability worlds together.Traces are the first step toward truly understanding cloud-native architectures. They capture a connected view of how requests travel through their applications, helping developers understand (in real time) how the different components in their system behave and interact with one another.With today’s announcement of tracing support, Promscale equips software engineers worldwide withobservability powered by SQLfor distributed systems, allowing them to unlock unprecedented insights about their systems.OpenTelemetry Tracing: Observability Powered by SQLWith tracing support, engineers can interrogate their observability data with arbitrary questions in SQL and get results back in real time to identify production problems faster and reduce mean time to repair (MTTR).OpenTelemetry tracing support in Promscale includes all these features:A native ingest endpoint for OpenTelemetry traces that understands the OpenTelemetry Protocol (OTLP) to ingest OpenTelemetry data easilyTrace data compression and configurable retention to manage disk usageFull SQL superpowers to interrogate your trace data with arbitrary questionsA fully customizable, out-of-the-box, and modern application performance management (APM) experience in Grafana using SQL queries on OpenTelemetry traces, to get new insights immediately after deploymentOut-of-the-box integrations to visualize distributed traces stored in Promscale using Jaeger and Grafana, so you don’t have to learn new tools or change existing workflowsSupport for ingesting traces in Jaeger and Zipkin formats via the OpenTelemetry Collector, so you can also benefit from all these new capabilities even if you’re not yet using OpenTelemetryCorrelation of Prometheus metrics and OpenTelemetry traces stored in Promscale throughexemplarsas well as SQL queriesIntegration intobs, the observability stack for Kubernetes, so you can deploy all those new capabilities with a single command in your Kubernetes clusterKeep reading this blog post for more details on these capabilities and an introduction to powerful, pre-built Grafana dashboards that you can integrate into your Grafana instance. These dashboards will give you instant information on your distributed systems’ dependencies, helping you quickly identify (and correct) performance bottlenecks.You candeploy Promscale in your environment today—it’s 100 % free—or experiment with it byrunning our lightweight OpenTelemetry demo. To learn how to deploy a microservices application and a complete OpenTelemetry observability stack running on Promscale in only a few minutes,check our OpenTelemetry demo blog post for a step-by-step guide.Sending and Storing Trace Data in PromscaleA key advantage of OpenTelemetry is that it is vendor agnostic. This allows us to instrument our code with OpenTelemetry libraries and send the telemetry generated to one or multiple observability backends via exporters. OpenTelemetry also defines a line protocol (OTLP). All the OpenTelemetry language SDKs include an exporter to send telemetry data to OTLP-compliant backends. A number of other exporters have been built by the community and vendors to send the data to non-OTLP-compliant backends.Promscale has an endpoint that listens for OTLP over Google Remote Procedure Calls (by default on port 9202). So, it can ingest traces using the standard OpenTelemetry tools without needing a proprietary exporter.OpenTelemetry also includes the OpenTelemetry Collector, which is a component that allows it to receive telemetry in multiple formats (not just OTLP), process it, and export it via OTLP with the OpenTelemetry exporter or in other formats through proprietary exporters.Architecture diagram illustrating the core components of the OpenTelemetry collector, the inputs it accepts, and possible outputsTo send trace data from your application instrumented with OpenTelemetry to Promscale, you have two options:You can configure the OTLP exporter of the OpenTelemetry SDK you used to instrument your application to send traces directly to Promscale.You can configure the OTLP exporter of the OpenTelemetry SDK to send data to the OpenTelemetry Collector and then from the OpenTelemetry Collector to Promscale.We advise you to use the OpenTelemetry Collector for better performance because it can send data to Promscale in larger batches, which speeds up ingestion. Also, if you want to send data to another backend, you can change the configuration in one place.See our documentation for more details.You can also easily sendJaegerandZipkintraces to Promscale through the OpenTelemetry Collector by configuring it to receive traces in those formats and convert them to OpenTelemetry.Promscale supports the full OpenTelemetry trace schema. It stores and queries all the data, including resources, spans, events, and links with their corresponding attributes. Promscale stores OpenTelemetry traces in TimescaleDB using an optimized database schema and leverages TimescaleDB’s compression capabilities to reduce disk space requirements by up to 10-20x. Theretention period is configurableand is set to 30 days by default.Querying Traces With SQLTo make querying trace data with SQL easier, Promscale exposes database views that make querying spans, events, and links (with all their corresponding attributes) as easy as querying single relational tables.For example, the following query returns the average response time (i.e., latency) in milliseconds for requests to each service in the last five minutes:SELECT
    service_name AS ""service"",
    AVG(duration_ms) as ""latency (ms)""
FROM ps_trace.span s
WHERE 
   start_time > NOW() - INTERVAL '5m'
   AND 
   (parent_span_id IS NULL
   OR
   span_kind = 'SPAN_KIND_SERVER'
   )
GROUP BY 1
ORDER BY 1 ASC;ps_trace.spanis the span view. The conditionparent_span_id IS NULL OR span_kind = 'SPAN_KIND_SERVERensures that only root spans (i.e., those that have no parent span) or spans that represent a service receiving a request from another service (i.e., this is whatkind serverindicates) are selected. Finally, we compute the average of the duration of those spans across the last five minutes for each service.This is the result of visualizing the query above in Grafana with a Table panel:Latency in milliseconds for requests to each service in the last five minutesBecause traces are represented as a hierarchy of spans, the ability to use SQL joins, aggregates, and recursive queries to perform sophisticated analysis on trace data is extremely valuable. We’ll say it again: observability is essentially an analytics problem, and SQL is the most popular tool for analytics.Imagine that we notice our customers are experiencing prolonged response times from a specific API endpoint we offer. When we receive a request to that API endpoint, there are many operations in multiple microservices that are called to fulfill that request. Identifying where the bottleneck is in a distributed environment is not trivial. How could we quickly identify where most of the time is effectively being spent?Note that there are a couple of important considerations to take into account.First, the microservices and operations involved in requests to that API endpoint could also be involved in many other requests. We only want to measure the performance of those service operationswhen they are called as part of a request to that API endpointand not when they are called because of requests to other endpoints.Secondly, we want to look at the effective execution time, that is, the time when a service operation is actually doing some processing and not just waiting for the response from another service operation (within the same service or another service). The graphic below illustrates the problem.In that particular trace, what is the operation that is consuming the most time? Well, if we just look at the duration of each span, the initial span (i.e., the root span) will always take more time since it includes all other spans. But that’s useless because it doesn’t help us identify the bottleneck. To compute the effective execution time of an operation, we need to subtract its duration from the duration of all child spans. In our example, the effective execution time of Operation A is t1 + t2 + t6, which is shorter than t4, which is the effective execution time of Operation C. Therefore, Operation C is the main bottleneck in this request.The root span can be broken down into child spans—the duration of each child span will give us clues on which operation may be causing a bottleneckNote that if we used parallelization or asynchronous operations, we would need to do this differently, but for the sake of simplicity, we’ll assume operations are sequential.To address those two problems, we need to take several steps:Compute the duration of each downstream service and operation only when they are being called as part of a request to the slow API endpointSubtract to each of those the duration of child spansAggregate the results by service and operationTo do this in SQL, we use a recursive query that uses the following syntax:Syntax of a recursive query in SQLThis is how it works.It first runs the initial query, which returns an initial set of results, and then runs the recursive query against that initial set of results to get another set of results. After that, it reruns the recursive query on that set of results and continues doing so until the recursive query returns no results.Then, it takes all the different results from all the individual queries and concatenates them. That’s what the UNION does.And finally, it runs the final query against all those concatenated results—that is what the query returns.The following query is the one we can run to compute the effective execution time of each operation over the last 30 minutes. In this case, the API endpoint we are analyzing is thegenerator.generateoperation (span_nameis the operation’s name) of thegeneratorservice.WITH RECURSIVE x AS
(
    -- initial query
    SELECT
        s.trace_id,
        s.span_id,
        s.parent_span_id,
        s.service_name,
        s.span_name,
        s.duration_ms - coalesce(
        (
            SELECT sum(z.duration_ms)
            FROM ps_trace.span z
            WHERE s.trace_id = z.trace_id
            AND s.span_id = z.parent_span_id
            AND z.start_time > NOW() - INTERVAL '30 minutes'
        ), 0.0) as effective_duration_ms
    FROM ps_trace.span s 
    WHERE s.start_time > NOW() - INTERVAL '30 minutes'
    AND s.service_name = 'generator'
    AND s.span_name = 'generator.generate'

    UNION ALL
    -- recursive query
    SELECT
        s.trace_id,
        s.span_id,
        s.parent_span_id,
        s.service_name,
        s.span_name,
        s.duration_ms - coalesce(
        (
            SELECT sum(z.duration_ms)
            FROM ps_trace.span z
            WHERE s.trace_id = z.trace_id
            AND s.span_id = z.parent_span_id
            AND z.start_time > NOW() - INTERVAL '30 minutes'
        ), 0.0) as effective_duration_ms
    FROM x
    INNER JOIN ps_trace.span s
    ON (x.trace_id = s.trace_id
    AND x.span_id = s.parent_span_id
    AND s.start_time > NOW() - INTERVAL '30 minutes')
    
)
-- final query
SELECT
    service_name,
    span_name,
    sum(effective_duration_ms) as total_exec_time
FROM x
GROUP BY 1, 2
ORDER BY 3 DESCThese are the key things to note in this query:The recursive query syntaxThe initial and the recursive queries compute the effective duration of a span by using a subquery to sum the duration of all child spans. The latter is then subtracted from the duration of the span (coalesce is used to return 0 in case the span has no child spans and the subquery returns NULL)s.duration_ms - coalesce(
(
	SELECT sum(z.duration_ms)
	FROM ps_trace.span z
	WHERE s.trace_id = z.trace_id
	AND s.span_id = z.parent_span_id
	AND z.start_time > NOW() - INTERVAL '30 minutes'
), 0.0) as effective_duration_msThe recursive query uses a join to traverse the downstream spans by selecting the child spans. Since the recursive query runs again and again on the results, applying the recursive query to the previous results ends up processing all downstream spans across all traces that originate in our API endpointINNER JOIN ps_trace.span s
ON (x.trace_id = s.trace_id
AND x.span_id = s.parent_span_id
AND s.start_time > NOW() - INTERVAL '30 minutes')The final query aggregates all the individual execution times by service and operationThis is the result of the query in Grafana using the Pie chart panel. It quickly points out therandom_digitoperation of thedigit serviceas the main bottleneck.Pie chart in Grafana showing the total execution time of each operation. With such a view, we can clearly identify that therandom_digitoperation of thedigitservice is the main bottleneckA Modern APM Experience Integrated Into GrafanaAn overview of the Application Performance Monitoring dashboards for Promscale in Grafana (right-click on ""Open Link in New Tab"" for a better view)Since it is directly integrated into your Grafana instance, you don’t need to set up a new tool or learn a new user interface. Additionally, the Grafana dashboards can be updated, which means you can customize them and extend them to meet your specific needs.The new APM experience within Grafana consists of six dashboards linked to each other:[1] Overview: provides a general overview of the performance across all services to surface the most common problems that could require your attention.[2] Service Details:provides a detailed view of the performance of a service to quickly identify specific problems in that service related to throughput, latency, and errors.[3] Service Map:a real-time automatic map of how your services communicate to identify dependencies and quickly validate their implementation.[4] DownstreamDependencies:a real-time detailed node graph with all the downstream services and operations across all selected service and operation traces. This helps you troubleshoot in detail how downstream services and operations impact the performance of an upstream service.[5] UpstreamDependencies:a real-time detailed node graph with all the upstream services and operations across all selected service and operation traces. This helps you quickly identify the potential blast radius of an issue in a specific service and operation and determine which upstream service and operation are causing problems (like a sudden increase in requests impacting performance) on a downstream service.Note: Some of these dashboards use the Node graph panel, which was introduced in recent versions of Grafana. It’s a beta version, so it may break. It worked for us in Grafana 8.5.1.These dashboards are available onGitHub(filename starts with apm-). Checkour documentationfor details on how to import the dashboards into Grafana.Observability Is an Analytics ProblemOpenTelemetry, a CNCF standard for instrumentation, makes telemetry data easier to collect and universally available. Traces, in particular, hold a treasure of valuable information about how distributed systems behave. We need powerful tools to analyze the data to get the most value from tracing. Observability is essentially an analytics problem.SQL is the lingua franca of analytics. When applied to trace data, it helps you unlock new insights to troubleshoot production issues faster and proactively improve your applications. Get started now withPromscale on Timescale (free 30-day trial, no credit card required)orself-hostfor free.Promscale is an observability backend built on the rock-solid foundation of PostgreSQL and TimescaleDB. With the new support for OpenTelemetry traces and integrations with the visualization tools you use and love, you have full SQL superpowers to solve even the most complex issues in your distributed systems. You can install Promscale for freehere.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/observability-powered-by-sql-understand-your-systems-like-never-before-with-opentelemetry-traces-and-postgresql/
2023-09-05T13:00:58.000Z,A Deep Dive Into PostgreSQL Vacuum Monitoring With BPFtrace,"Observability enables engineers to quickly troubleshoot and fix issues in production. For those using the Linux kernel, theextended Berkeley Packet Filter (eBPF) technologyallows application monitoring with minimal overhead. For example, you can useUProbesto trace the invocation and exit of functions in programs.Modern database observability tools (e.g.,funccount) are built on top of eBPF. However, these fully flagged tools are often written in C and Python and require some development effort when a ""quick and dirty"" solution for a particular observation is sometimes more than sufficient. However, withBPFtrace, a high-level tracing language for Linux eBPF, users can create eBPF programs with only a few lines of code.In this tutorial, we develop a simple BPFtrace program to observe the execution of vacuum calls in PostgreSQL and measure and print their execution times.The EnvironmentPostgreSQLusesvacuum operationsto reclaim space from dead (e.g., updated or deleted) tuples.For more information on how this process works and why it is an important one to monitor, read this article.In this tutorial, we will trace the vacuum calls and determine the required time per table for the vacuum operations. We will use a PostgreSQL 14 server, with the  binary is located at/home/jan/postgresqlsandbox/bin/REL_14_2_DEBUG/bin/postgres.In addition, the examples are executed in a database with these two tables:CREATE TABLE testtable1 (
   id int NOT NULL,
   value int NOT NULL
);

CREATE TABLE testtable2 (
   id int NOT NULL,
   value int NOT NULL
);❗Note:Depending on the used C compiler and applied optimizations, the symbols of internal functions (i.e., as static declared) may not be visible. In this case, you can not use UProbes to trace the function invocations. There are two possible solutions to address this issue: (1) remove the static modifier from the function declaration and recompile PostgreSQL, or (2) create a completedebug buildof PostgreSQL.Tracing PostgreSQL Vacuum Operations UsingfunclatencyLet’s explore the existing solutions before developing our tool to trace the vacuum operations.The toolfunclatencyis available for most Linux distributions (on Debian, it’s part of the packagebpfcc-tools and renamed to funclatency-bpfcc) and allows it to trace a function enter and exit and measure the function latency (i.e., the time it takes a function to complete).In PostgreSQL, the functionvacuum_relis invoked when a vacuum operation on a relation is performed. To trace these function calls withfunclatency-bpfcc, you need to provide the path of the PostgreSQL binary and the function name. For instance:‌$ sudo funclatency-bpfcc -r /home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel

Tracing 1 functions for ""/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel""... Hit Ctrl-C to end.Afterward, an eBPF program is loaded into the Linux kernel, and a UProbe is defined and observes the beginning of the function call while another UProbe observes its exit. The latency between these two events is measured and stored.To execute some vacuum operations, we perform the following SQL statement in a second session:database=# VACUUM FULL;
VACUUM FULL‌This SQL statement triggers PostgreSQL to perform a vacuum operation of all tables of the current database—this takes some time. After the vacuum operations are done, thefunclatency-bpfccprogram can be stopped (by executing CTRL+C), ending the observation of the binary and showing the recorded execution times on the terminal.$ sudo funclatency-bpfcc -r /home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
[...]
^C
Function = b'vacuum_rel' [876997]
     nsecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 0        |                                        |
        64 -> 127        : 0        |                                        |
       128 -> 255        : 0        |                                        |
       256 -> 511        : 0        |                                        |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 0        |                                        |
      2048 -> 4095       : 0        |                                        |
      4096 -> 8191       : 0        |                                        |
      8192 -> 16383      : 0        |                                        |
     16384 -> 32767      : 0        |                                        |
     32768 -> 65535      : 0        |                                        |
     65536 -> 131071     : 0        |                                        |
    131072 -> 262143     : 0        |                                        |
    262144 -> 524287     : 0        |                                        |
    524288 -> 1048575    : 0        |                                        |
   1048576 -> 2097151    : 0        |                                        |
   2097152 -> 4194303    : 0        |                                        |
   4194304 -> 8388607    : 2        |*                                       |
   8388608 -> 16777215   : 13       |***********                             |
  16777216 -> 33554431   : 44       |****************************************|
  33554432 -> 67108863   : 7        |******                                  |
  67108864 -> 134217727  : 1        |                                        |

avg = 22765358 nsecs, total: 1525279002 nsecs, count: 67

Detaching...‌The output contains the information that the functionvacuum_relwas called 67 times, and the average function time is22765358 nsecs.In addition, a histogram of the function latency is printed, providing a lot of helpful information. However, while we now know the number and duration of vacuum calls, we still don’t know the duration of the individual vacuum calls for each relation.This is something that this tool does not support because it does not evaluate the function parameters (e.g., the Oid of relation that the current function invocation should vacuum). But this is something we can do withbpftrace.Using BPFtrace To Trace VACUUM FULL EntriesLet’s start with a very simple BPFtrace program that prints a line once thevacuum_relfunction is invoked in the PostgreSQL binary.bpftraceis called with the eBPF program that should be loaded into the Linux kernel. The eBPF programs that are passed to BPFtrace have the followingsyntax:<probe1> {
        <Actions>
}

[...]

<probeN> {
        <Actions>
}‌The syntax to define a UProbe on a binary or library is:uprobe:library_name:function_name[+offset]. For instance, to define a UProbe on the function invocation ofvacuum_relin the binary/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgresand print the lineVacuum started, you can use the following BPFtrace call:$ sudo bpftrace -e '
uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel {
    printf(""Vacuum started\n"");
}
'

Attaching 1 probe...
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
[...]As soon as theVACUUM FULLSQL statement in PostgreSQL is executed in another terminal session, the program starts to print the message on the screen. This is a good start, but we still have less information available than output by the existing toolfunclatency-bpfcc. We're missing the latency of the function calls.Tracing Vacuum Function Returns and LatencyTo measure the execution time of the function invocations, we need two things:Define a second Uprobe that is invoked when the function observed returns; this can be done by using a URetProbe.Calculate the difference between the two UProbe events.A URetProbe in BPFtrace can be defined using the same syntaxuretprobe:binary:functionas the one used for the UProbe. In addition, BPFtrace allows it to create variables like associative arrays.We use such an array to capture the start time of a function invocation@start[tid] = nsecs;. The array's key is the ID of the current thread:tid. So, multiple threads (and processes like in our case with PostgreSQL) can be traced simultaneously without overriding the last function invitation start time.In the URetProbe we take the current time and subtract the time of the function invocation (nsecs - @start[tid])) to get the time the function call needs. We also use a function predicate (/@start[tid]/) to let BPFtrace know that we only want to execute the function body of the URetProbe as soon as this array value is defined.Using this predicate, we prevent handling a function return without seeing the function enter before (e.g., we start theBPFtrace program in the middle of a running function call, and we get only the URetProbe invocation for this function call).❗Note:It is not guaranteed that BPFtrace will deliver and process the eBPF events in order. Especially when a function call is short, and we have a lot of function invocations, the events could be processed out-of-order (e.g., we see two function enter events followed by two function return events). In this case, function latency observations with BPFtrace become imprecise. To avoid this, we use VACUUM FULL calls instead of vacuum calls. These calls aremuch more expensivesince they rewrite the table. Therefore, they take longer and can be reliably observed by BPFtrace.$ sudo bpftrace -e '
uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf(""Performing vacuum\n"");
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf(""Vacuum call took %d ns\n"", nsecs - @start[tid]);
        delete(@start[tid]);
}
'After running this BPFtrace call and executingVACUUM FULLin a second session, we see the following output:Attaching 2 probes...
Performing vacuum
Vacuum call took 37486735 ns
Performing vacuum
Vacuum call took 16491130 ns
Performing vacuum
Vacuum call took 32443568 ns
Performing vacuum
Vacuum call took 17959933 ns
[...]‌For each call of thevacuum_relfunction in PostgreSQL, we measure the time the vacuum operation needs. However, it would also be convenient to capture the Oid or the name of the relation vacuumed by the current vacuum operation. This requires the handling of the function parameters of the observed function.Handle Function ParametersThe functionvacuum_relhas the following signature in PostgreSQL 14. The first parameter is theOid(anunsigned int) of the processed relation. The second parameter is aRangeVarstruct, whichcouldcontain the name of the relation. The third parameter is aVacuumParamsstruct, which contains additional parameters for the vacuum operation, and the last parameter is aBufferAccessStrategy, which defines the access strategy of the used buffer.static bool vacuum_rel(Oid relid,
        RangeVar *relation,
        VacuumParams *params,
        BufferAccessStrategy bstrategy 
)BPFtrace allows it to access the function parameter using the keywordsarg0,arg1, …,argN. To include the Oid in the output of our logging, we need only to print the first parameter of the function.$ sudo bpftrace -e '

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf(""Performing vacuum of Oid %d\n"", arg0);
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf(""Vacuum call took %d ns\n"", nsecs - @start[tid]);
        delete(@start[tid]);
}
'When theVACUUM FULLoperation is executed again in a second terminal, the output looks as follows:Attaching 2 probes...
[...]
Performing vacuum of Oid 1153888
Vacuum call took 37486734 ns
Performing vacuum of Oid 1153891
Vacuum call took 49535256 ns
Performing vacuum of Oid 2619
Vacuum call took 39575635 ns
Performing vacuum of Oid 2840
Vacuum call took 40683526 ns
Performing vacuum of Oid 1247
Vacuum call took 14683600 ns
Performing vacuum of Oid 4171
Vacuum call took 20587503 ns‌To determine which Oid belongs to which relation, the following SQL statement can be executed:blog=# SELECT oid, relname FROM pg_class WHERE oid IN (1153888, 1153891);
   oid   |  relname   
---------+------------
 1153888 | testtable1
 1153891 | testtable2
(2 rows)The result shows that Oids1153888and1153891belong to the tablestesttable1andtesttable2, which we have created in one of the first sections of this article. These values belong to our test environment. In your environment, different OIDs might be shown.Handle Function Struct ParametersSo far, we have processed simple parameters withbpftrace(like Oids, which are unsigned integers). However, many parameters in PostgreSQL are C data structs. Furthermore, these structs can be handled in BPFtrace programs as well.The second parameter of thevacuum_relfunction is a RangeVar struct. This struct isdefined in PostgreSQL 14as follows:typedef struct RangeVar
{
	NodeTag	type;
	char *catalogname;
	char *schemaname;
	char *relname;
	[...]
}To process the struct, the following BPFtrace program can be used. Please note, that the internalNodeTagdata type of PostgreSQL is replaced by a simple int.TheNodeTagdata type is anenum. Enums are backed by the integer data type in C. To handle this enum correctly, we could (1) also copy the enum definition into the eBPF program, or (2) we could replace it with a data type of the same length.To keep the BPFtrace program simple, the second option is used here. The next three struct members are char pointer, which contains thecatalogname, the schema, and the name of the relation. Theschemanameand therelnameare the fields we are interested in. The struct contains more members, but these members are ignored to keep the example clear.$ sudo bpftrace -e '
struct RangeVar
{
	int type;
	char *catalogname;
	char *schemaname;
	char *relname;
};

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf(""[PID %d] Performing vacuum of Oid %d (%s.%s)\n"", pid, arg0, str(((struct RangeVar*) arg1)->schemaname), str(((struct RangeVar*) arg1)->relname));
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf(""[PID %d] Vacuum call took %d ns\n"", pid, nsecs - @start[tid]);
        delete(@start[tid]);
}
'After the struct is defined, the members of the struct can be accessed as in a regular C program. For example:((struct RangeVar*) arg1)->schemaname. In addition, we also print the process ID (PID) of the program that has triggered the UProbe. This allows it to identify the process that has performed the vacuum operation.When running the following SQL statements in a second terminal:VACUUM FULL public.testtable1;
VACUUM FULL public.testtable2;The BPFtrace program shows the following output:Attaching 2 probes...
[PID 616516] Performing vacuum of Oid 1153888 (public.testtable1)
[PID 616516] Vacuum call took 23683600 ns
[PID 616516] Performing vacuum of Oid 1153891 (public.testtable2)
[PID 616516] Vacuum call took 24240837 nsThe table names are extracted from theRangeVardata structure and shown in the output. However, this data structure is not always populated by PostgreSQL. The data structure might be empty when runningVACUUM FULLwithout specifying a table name. Therefore, we use two single invocations with explicit table names to force PostgreSQL to populate this data structure.Optimizing the BPFtrace Program Using MapsThe BPFtrace programs we have developed so far use one or moreprintfstatements directly. Aprintfcall is slow and reduces the throughput the BPFtrace program can monitor.This can be optimized by storing the data in map that is printed when BPFtrace is stopped. This will postpone the printf calls until observation is done. To do this, we introduce three new maps@start,@oid, and@vacuum. The first two maps are populated in the UProbe event of thevacuum_relfunction. The map@startcontains the time when the probe is triggered, and the map@oidcontains the Oid of the parameter function.When the function returns, and the URetProbe is triggered, the@vacuummap is populated. The key is the Oid, and the value is the needed time to perform the vacuum operation. Also, the keys of the first two maps are removed.When BPFtrace exits (i.e., by pressing CRTL+C), all populated maps are printed automatically. By using these three maps (@start,@oid,@vaccum), we have separated the actual monitoring from the output, the expensiveprintffunction is called after the monitoring is done.In addition, in the following program, we use the two functionsBEGINandENDthat are called by BPFtrace when the observation begins and ends.$ sudo sudo bpftrace -e '

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        @start[tid] = nsecs;
        @oid[tid] = arg0;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{

        @vacuum[@oid[tid]] = nsecs - @start[tid];
        delete(@start[tid]);
        delete(@oid[tid]);

}

BEGIN
{
        printf(""VACUUM calles are traced, press CTRL+C to stop tracing\n"");
}

END 
{
        printf(""\n\nNeeded time in ns to perform VACUUM FULL per Oid\n"");
}
'After BPFtrace is started, the first message is printed. After the program is stopped, the second message is printed. Furthermore, the content of the@vacuummap is printed. For each Oid, the needed time for the vacuum operations is shown.VACUUM calls are traced, press CTRL+C to stop tracing
^C

Needed time in ns to perform VACUUM FULL per Oid

@vacuum[1153888]: 7526823
@vacuum[1153891]: 8462672
@vacuum[2613]: 10764797
@vacuum[2995]: 11429589
@vacuum[6102]: 11436539
@vacuum[12801]: 14373934
@vacuum[6106]: 14396012
@vacuum[3118]: 14507167
@vacuum[3596]: 14695385
@vacuum[12811]: 14871237
@vacuum[3429]: 15106778
@vacuum[3350]: 15158742
@vacuum[2611]: 15432053
@vacuum[3764]: 15534169
@vacuum[2601]: 16055863
@vacuum[3602]: 16128624
@vacuum[2605]: 16405419
@vacuum[2616]: 16914195
@vacuum[3576]: 17003920
[...]ConclusionThis article provides a brief overview of BPFtrace. To trace the function latency of PostgreSQL vacuum calls, we used the toolfunclatency-bpfcc.Additionally, we utilized BPFtrace to create a tool that allows for more in-depth observation of the calls. Our BPFtrace script also takes into account the parameters of the PostgreSQLvacuum_relfunction, enabling us to monitor the vacuum time per relation.And speaking of vacuum,check out this blog post to learn more about transaction ID wraparound exhaustion and how to avoid it in PostgreSQL databases.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/using-bpftrace-to-trace-postgresql-vacuum-operations/
2022-05-10T13:08:07.000Z,A Deep Dive Into OpenTelemetry Metrics,"This is the second post of a three-part series about metrics. First,we deep-dived into the four types of Prometheus metrics;now, we're examining how metrics work in OpenTelemetry, and finally, we will put the two together—explaining the differences, similarities, and integration between the two.OpenTelemetryis an open-source observability framework for infrastructure instrumentation hosted by theCloud Native Computing Foundation (CNCF). The project gained a lot of momentum with contributions from all major cloud providers (AWS, Google, Microsoft) as well as observability vendors (including Timescale) to the point it becamethe second-highest ranked CNCF project by activity and contributors, only coming second to Kubernetes itself.OpenTelemetry aims todefine a single standard across all types of observability data(which it refers to as signals), including metrics, logs, and traces. Through a collection of tools, libraries, APIs, SDKs, and exporters, OpenTelemetry radically simplifies the process of collecting signals from your services and sending them to the backend of your choice, opening the doors of observability to a wider range of users and vendors.When designing the specification of the OpenTelemetry metrics, threedesign goalswere embraced:To provide the ability to connect metrics to other types of observability data—either directly between traces and metrics withExemplars, or indirectly, by providing the same associated metadata that logs and traces enjoy withBaggageandContext.To allow easy migration fromOpenCensusmetrics to OpenTelemetry metrics.To provide full support for other major metrics implementations where possible. Prometheus andStatsDare specifically targeted to have full support, with users who convert to OpenTelemetry seeing similar results to using their native clients.OpenTelemetry provides a Collector which can be used to reaggregate and redirect metric data, often being used to create signal pipelines. As OpenTelemetry doesn’t provide a backend implementation (its concern is creating, collecting, and sending signals), the data will flow to another system or systems for storage and eventual querying.The reason why OpenTelemetry can sometimes feel complicated it's because it can be used to model many different signal implementations. In this blog post, we will focus onthesurface area—the elements that developers are likely to encounter when using metrics in OpenTelemetry.OpenTelemetry MetricsOpenTelemetry metrics are slightly different fromPrometheus metrics, allowing much more flexibility in the way they can be transformed in the collection path, as well as supporting many export types including push and pull options. Due to this flexibility, many existing metric systems can be modeled with OpenTelemetry without the loss of semantics or fidelity, which makes it the perfect metric system for interoperability.OpenTelemetry has three models:The Event model, in which you create metrics as a developerThe Stream model, which OpenTelemetry uses for transportThe Timeseries model, which OpenTelemetry uses for storageThe metric types discussed in this post are all part of the Event model, and the transformations are part of the conversion from the Event Model to the Stream Model. (As a developer, you don’t really need to worry about the models—but having a basic understanding helps.)The OpenTelemetry SDK allows us to:Reduce the number of metrics being transmitted through temporal aggregation (changing the resolution)Reduce the number of metrics being transmitted through spatial aggregation (removing unwanted attributes)Change from a cumulative representation (which Prometheus uses) to a delta representation (expressing the change between values, not the absolute measurements)OpenTelemetry metrics work by using the globalMeterProviderto create aMeterand associating it with one or moreInstruments, each of which is used to create a series ofMeasurements. These Measurements are aggregated inViewsto a metric. The metrics are then observed and emitted by the combination of aMetric ReaderandMetric Exporter(which could be pull or push).Diagram illustrating the elements of the MeterProvider in OpenTelemetryInstrument Measurements are what we create or observe in our applications, and Metrics express the current aggregated value of that Instrument that we share with consumers of our observability data.OpenTelemetry allows attributes (labels) to be attached to metrics in several ways. The most common are:From any attached Resources, which might hold labels defining the host. For example, a Kubernetes pod or a virtual machineFrom the current Context, which will be attached to all synchronous InstrumentsOn the Measurement itselfMeasurements to MetricsMeasurements can be created very quickly, especially when synchronous, which can rapidly overwhelm a metric pipeline. To combat this, Prometheus uses a pull mechanism with a scrape interval, while OpenTelemetry addresses the issue in the collection path by attaching an aggregating view to each Instrument, then passing the data to a MetricReader which observes them, and a MetricExporter which outputs them:From measurements to metrics: diagram illustrating the metrics’ collection path in OpenTelemetryThe MetricReader is responsible for attaching the default view if no views are present for an Instrument and also for defining MetricExporters, which will send the values onward. It will also change the temporality of the metrics from the default Cumulative (the new value is added to the last value, which is the same as we see in Prometheus) to Delta (the metric value is the difference between the old and new values).Each Instrument is associated with a MetricReader, which is responsible for attaching the default views if no other views are defined for an Instrument. In addition, it also defines the temporality—potentially switching from the default Cumulative (the current value is added to the previous values, as used in Prometheus) to Delta (the difference between the current and last values is reported, reducing the client overhead when calculating rates).The combination of the MetricReader and the MetricExporter is responsible for how the data is sent downstream. A very popular approach is to use the PeriodicExportingMetricReader with the OTLPMetricExporter to sample the metric values every period (60 seconds by default) and send them to an Opentelemetry Collector (which would use another Exporter) for further processing. Many other Exporters are available for various languages.Some popular ones are:Prometheus Metric Exporter:a pull-based exporter which Prometheus clients can scrape.Prometheus Remote Write Exporter:a push-based exporter which sends metrics via the Prometheus remote write protocol.OTLPMetricExporter:it can push metrics to any device which understands the OpenTelemetry protocol.ConsoleMetricExporter:it is used to write debug messages to the console.In Python, initializing OpenTelemetry metrics and attaching a default MetricReader and MetricExporter (that will send metrics to the local OpenTelemetry Collector) would look like this:from opentelemetry._metrics import get_meter_provider, set_meter_provider
from opentelemetry.exporter.otlp.proto.grpc._metric_exporter import (
    OTLPMetricExporter,
)
from opentelemetry.sdk._metrics import MeterProvider
from opentelemetry.sdk._metrics.export import PeriodicExportingMetricReader

exporter = OTLPMetricExporter(insecure=True)
reader = PeriodicExportingMetricReader(exporter)
provider = MeterProvider(metric_readers=[reader])
set_meter_provider(provider)Instruments and Emitting MeasurementsOpenTelemetry provides six types of Instruments that we can capture Measurements with. They can be grouped into two categories: synchronous and asynchronous. Each Instrument can emit Measurements, and each Instrument can be associated with attributes.Synchronous instruments are implemented in application code in a similar way to Prometheus metrics, by inserting code into applications that will update a value each time it is executed. They can be associated with the current Context (which helps to describe the current application state).Asynchronous instruments register a callback function and only emit values when they are being observed. For example, an asynchronous instrument could be registered to report the value of a sensor every 10 seconds. These instruments can not be associated with the current Context, as they are external to the main application—instead of being called as the main program runs, they observe signal data as requested from their watcher. In some ways, they are similar to the Prometheus monitoring of a non–instrumented application via an exporter.All instruments are created with a name, a description, and a unit of measurement (which must follow theinstrument unit rules). Asynchronous instruments also specify the callback function, which is called to observe Measurements.The OpenTelemetry instrument types that developers can use are shown in the table below. (Confusingly, the suggested name that languages present to a user is not the same as the measurement name, with Observable being used in place of Asynchronous. I.e., observable_counter.)Table outlining the name and characteristics of the OpenTelemetry instrument typesNow, let’s expand into each instrument type.Counter / Asynchronous CounterA Counter is a synchronous instrument that is always increasing—it’s monotonic, and only accepts non–negative values. Unsurprisingly, it’s the same as the Prometheus Counter. Counters work by receiving increment or delta values.When using a Counter, anaddoperation will be available in the language SDK, which must be provided with the non–negative number to increment the Counter by, along with an optional set of attributes to be attached. These attributes are similar to Prometheus labels.An Asynchronous Counter differs by operating via callback rather than theaddfunction. When the instrument is observed, the callback is executed and will pass back one or more measurements expressed as absolute values (not delta values). Once the values have been passed, they will be changed into delta values internally.For example, you could implement an Asynchronous Counter that reports on the amount of CPU time the application has consumed since its start. This information would be extracted from the operating system in the callback and returned. Several values, one for each CPU or thread, could be returned at once. These measurements are always expected to be summable across attributes in a meaningful way (in this case, to get the total CPU time used for the system).Please note that as the Python SDK is not yet stable, we need to import _metrics rather than metrics for the code examples in this post. There may also be some breaking changes in the future, we will keep this post up to date as things progress. The current examples were written with the OpenTelemetry Python SDK v1.11.1.In Python, an example of creating and using a Counter and Asynchronous Counter would look like this:from opentelemetry._metrics import get_meter_provider
from opentelemetry._metrics.observation import Observation

meter = get_meter_provider().get_meter(""otel-demo"")

# Counter
counter = meter.create_counter(""counter"")

# This adds 100 to the total (a delta value)
counter.add(100,{""app"": ""timescale""})

# Callback function for Async counter
def observable_counter_func() -> [Observation]:
    # This reports that the current value is 10, which will be
    # converted to a delta internally
    return [Observation(10, {""app"": ""timescale""}]

# Async Counter
observable_counter = meter.create_observable_counter(
    ""observable_counter"", [observable_counter_func]
)UpDownCounter / Asynchronous UpDownCounterAn UpDownCounter is a similar synchronous instrument to a Counter, but it allows negative delta values to be passed (it’s not monotonic). Where a Counter would be suited to represent the number of jobs that had been submitted, an UpDownCounter would be perfect to represent the current number of active jobs being processed (it can move up and down). It’s important to note that this is not used identically to a Gauge in Prometheus, as we are recording changes, not absolute values.An UpDownCounter presents anaddoperation that is identical to the Counter operation—with the exception that it accepts negative data values. Values correlated by attribute data are expected to be able to be summed.Unsurprisingly, an Asynchronous UpDownCounter provides a callback interface that returns one or more measurements, expressing each measurement as an absolute value which will be changed to a delta value internally.The OpenTelemetry spec notes that an UpDownCounter should not be used when the value being returned can easily be observed. In that case, an Asynchronous UpDownCounter should be used instead.In Python, an example of creating and using an UpDownCounter and Asynchronous UpDownCounter would look like this:from opentelemetry._metrics import get_meter_provider
from opentelemetry._metrics.observation import Observation

meter = get_meter_provider().get_meter(""otel-demo"")

# UpDownCounter
up_down_counter = meter.create_up_down_counter(""up_down_counter"")

# This adds 100, then removes 10 from the total (both delta values)
up_down_counter.add(100,{""app"": ""timescale""})
up_down_counter.add(-10,{""app"": ""timescale""})


# Callback function for Async counter
def observable_up_down_counter_func() -> [Observation]:
    # This reports that the current value is 10, which will be
    # converted to a delta internally
    return [Observation(10, {""app"": ""timescale""})]

# Async UpDownCounter, note the observable prefix
observable_up_down_counter = meter.create_observable_up_down_counter(
    ""observable_up_down_counter"", [observable_up_down_counter_func]
)HistogramA Histogram is a synchronous instrument which allows the recording of multiple values that are statistically relevant to each other. You would choose a Histogram when you don't want to analyze data points in isolation, but would rather generate statistical information about their distribution by tracking the number of values that fall in each predefined bucket, as well as the minimum and the maximum value (if configured to do so).Histograms have a single method that is exposed:record. Record takes a non–negative observation value and an optional set of attributes to be attached.You might select a Histogram when you are looking at HTTP response times—knowing the exact response time of each request as it happens might not be that useful (and is much more suited to trace data, which would expose the start and finish times of each request), but being able to report on the median response time and the amount of HTTP requests above the 95th percentile might be more interesting from a service-level perspective.It’s important to note that recording the measurements doesn’t create the Histogram; the default aggregation (Explicit Bucket Histogram Aggregation) does. When using a Histogram instrument, it’s important to make sure the buckets are also configured. The defaults are [ 0, 5, 10, 25, 50, 75, 100, 250, 500, 1000 ], and that’s not always ideal. You can see some examples of creating a view in the following paragraphs.In Python, an example of creating and using a Histogram would look like this:from opentelemetry._metrics import get_meter_provider
from opentelemetry._metrics.observation import Observation

meter = get_meter_provider().get_meter(""otel-demo"")

# Histogram
histogram = meter.create_histogram(""histogram"")

# This records a value of 100
histogram.record(100,{""app"": ""timescale""})Asynchronous GaugeAn Asynchronous Gauge is unique in the OpenTelemetry API in two ways: it does not have a synchronous counterpart, and is designed to represent values that do not make sense to sum, even if they share attribute data. An example of this would be the temperature in various rooms of a house. This is common data, but it does not make any sense to report it as a total value—you’d potentially want an average or maximum, but never a sum. This is a different approach to Prometheus, which encoded these types of requirements into the metric naming conventions. In OpenTelemetry, if you use an Asynchronous Gauge, you will not be able to aggregate it as with all other metric types.In the same manner, as all Asynchronous instruments, a callback is passed when creating an Asynchronous Gauge, which can return one or more (in this case completely discrete) measurements.In Python, an example of creating and using an Asynchronous Gauge would look like this:from opentelemetry._metrics import get_meter_provider
from opentelemetry._metrics.observation import Observation

meter = get_meter_provider().get_meter(""otel-demo"")


# Callback function for Async gauge
def observable_gauge_func() -> [Observation]:
    # This reports that the current value is 10
    return [Observation(10, {""app"": ""timescale""})]

# Async Gauge, note the observable prefix
observable_gauge = meter.create_observable_gauge(
    ""observable_gauge"", [observable_gauge_func]
)Views and AggregationsA View in OpenTelemetry defines an aggregation, which takes a series of measurements and expresses them as a single metric value at that point in time. As more measurements are created, the metric is continuously updated. If there is no View created for an Instrument, then a default aggregation is selected based on the Instrument type. Custom views can be targeted by Meter name, Instrument name, Instrument type, or with a wildcard.There are three aggregation types available in OpenTelemetry:Sum Aggregation:itsimply tracks the sum of the incoming measurements (respecting the monotonicity of the input Instrument).Last Value Aggregation:it tracks the last value reported.Explicit Bucket Histogram Aggregation:it tracks the number of measurements that fall into each configured bucket (which must be predefined when the View is created), and can track the minimum and maximum value.The following table defines the default aggregation for each Instrument type:Table outlining the default aggregation for each OpenTelemetry instrument typeThis Python code uses the ConsoleMetricExporter to write to the console, also changing the bucketing for all histogram Instruments:from opentelemetry._metrics import get_meter_provider, set_meter_provider
from opentelemetry.sdk._metrics import MeterProvider
from opentelemetry.sdk._metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk._metrics.export import ConsoleMetricExporter
from opentelemetry.sdk._metrics.aggregation import ExplicitBucketHistogramAggregation


exporter = ConsoleMetricExporter()
reader = PeriodicExportingMetricReader(exporter)
provider = MeterProvider(
    metric_readers=[reader],
    views=[View(
        instrument_name=""*”, 
        aggregation=ExplicitBucketHistogramAggregation(
            (1,20,50,100,1000)
    ))],
)
set_meter_provider(provider)Summing It UpIn this second part of our blog post series on metrics, we discussed the OpenTelemetry standard, focusing on its six instrument types: counters, asynchronous counters, UpDownCounters, asynchronous UpDownCounters, histograms, and asynchronous gauges.In the final blog post of the series, we will cover how this model compares to Prometheus, explain the differences, and share our personal recommendations on when to consider one or the other.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/a-deep-dive-into-open-telemetry-metrics/
2023-05-10T16:45:49.000Z,How to Fix Transaction ID Wraparound Exhaustion,"Timescale employsPostgreSQL contributorswho are working feverishly to mitigate a problem with PostgreSQL. That problem is commonly referred to as “transaction ID wraparound” and stems from design decisions in the PostgreSQL project that have been around for decades.Because this design decision was made so early in the project history, it affects all branches and forks of PostgreSQL, with Amazon RDS PostgreSQL, Greenplum, Netezza, Amazon Aurora, and many others suffering from transaction ID wraparound failures.In this article, the second in aseries of posts tackling PostgreSQL errors or issues, we’ll explain what transaction ID wraparound is, why it fails, and how you can mitigate or resolve it. But let’s start with a bit of PostgreSQL history.Transaction ID Wraparound (XID Wraparound)To fully understand the problem of transaction ID wraparound (or XID wraparound), a bit of history is in order. The idea of a transaction counter in PostgreSQL originated as a very simple answer to transaction tracking. We need to know the order in which transactions are committed to a PostgreSQL database, so let's enumerate them. What is the simplest way to give transactions a concept of order? That would be a counter. What is the simplest counter? An integer. Tada!So, form follows function, and we have an integer counter. Seems like an obvious, elegant, and simple solution to the problem, doesn't it?At first glance (and second and third, honestly), this rather simple solution stood up very well. Who would ever need more than 231(just over 2 billion) transactions in flight? That was an astronomical number for 1985.Since this is such a huge number, we should only need a single counter for the entire database cluster. That will keep the design simple, prevent the need to coordinate multiple transaction counters and allow for efficient (just four bytes!) storage. We simply add this small counter to each row, and we know exactly what the high watermark is for every row version in the entire cluster.This simple method is row-centric and yet cluster-wide. So, our backups are easy (we know exactly where the pointer is for the entire cluster), and the data snapshot at the beginning and end of our transaction is stable. We can easily tell within the transaction if the data has changed underneath us from another transaction.We can even play peek-a-boo with other transaction data in flight. That lets us ensure that transactions settle more reasonably, even if we are wiggling the loose electrical connectors of our transaction context a bit.We can stretch that counter quite a bit by making it a ring buffer. That is, we'llORthe next value to the end rather than add it there. That way, 231or 1 = 1. So, our counter can wrap around the top (231) and only becomes problematic when it reaches the oldest open transaction at the bottom.This ""oldest"" transaction is an upwardly moving number also, which then wraps around the top. So, we have the head (current transaction) chasing the tail (oldest transaction) around the integer, with 2,147,483,648 spaces from the bottom to the top. This makes our solution even look like a puppy dog, so now it's cute as well as elegant.The idea is that this would make the counteralmostinfinite, as the head should never catch the tail. At that point, who could possibly need more transactions than that? Brilliant!Transaction counters are obviously the way to go here. They just make everything work so elegantly.Explanation: The Plan in ActionFor many years, PostgreSQL raged forward with the XID wraparound transaction design. Quite a few features were added along the way that were based on this simple counter. Backups (pg_basebackupand its cousins), replication (both physical and logical), indexes, visibility maps, autovacuum, and defragmentation utilities all sprouted up to enhance and support this central concept.All of these things worked together brilliantly for quite some time. We didn't start seeing the stress marks in the fuselage until the hardware caught up with us. As much as PostgreSQL wants to turn a blind eye to the reality of the hardware universe, the time came upon us when systems had the capacity to create more than 231transactions at a time.High-speed ETL, queuing systems, IoT, and other machine-generated data could actually keep the system busy long enough that the counter could be pushed to its inherent limit.""Everybody has a plan until I punch them in the face.""—Mike TysonThese processes weren't exactly showstoppers, though. We came up with band-aids for much of it.COPYgot its own transaction context, reducing the load significantly. So didVACUUM.VACUUMsprouted the ability to just freeze transactions without having to do a full row cleanup. That made the tail move forward a bit more quickly. External utilities gained features, individual tables gainedVACUUMsettings so they could be targeted separately.Okay, that helped. But did it help enough? These features were never designed to fundamentally fix the issue. The issue is that size matters. But to be a bit more descriptive...Possible CausesHow big is big?In the early aughts, I was involved in building a data center for a private company. We spent some $5M creating a full-featured data center, complete with Halon, a 4K generator, Unisys ES7000, and a Clarion array. For the sake of our XID wraparound article, I'll focus on the Clarion array. It cost just a bit over $2M and could hold 96 drives for a whopping total of 1.6 TB! In 2002, that was incredible.It doesn't seem so incredible now, does it? Kinda disappointing even. A few weeks ago, I needed an additional drive for my home backup unit. As I was walking through a Costco, I absent-mindedly threw a 2 TB drive into my cart that retailed for $69. It wasn't until I got home and was in the middle of installing it that it dawned on me how far we've come in the storage industry.Some of the young whippersnappers don't even care about storage anymore. They think the ""cloud"" storage is effectively infinite.They're not wrong.To bring this around to PostgreSQL, tables with 2M rows were a big deal in 2002. Now that's not even on the radar of ""big data."" A VLDB (very large database) at the time was 2 TB. Now it's approaching 1 PB.""A lot"" of transactions in 2002 was 2M. Now, I would place that number at somewhere around 2B. Oops. Did I just say 2B? Isn't that close to the same number I said a few paragraphs ago was the limit of our transaction space? Let me see, that was 231, which is 2,147,483,648.Ouch.How to Resolve Transaction ID Wraparound FailureTo be fair, not everybody has this problem. 2,147,483,648 is still a really big number, so a fairly small number of systems will ever reach this limit, even in the transaction environment of 2023.It also represents the number of transactions that are currently in flight, as the autovacuum process will latently brush away transaction counters that are no longer visible to the postmaster (pg_stat_activity).  But if the number of phone calls to consultants is any indication, this limitation is nonetheless becoming quite an issue. It certainly isn't going away any time soon.Everybody in the PostgreSQL ecosystem is painfully aware of the limitation. This problem affects more than just the core of PostgreSQL, it affects all of the systems that have grown around it also. Do you know what it also affects? All the PostgreSQL-based databases, such as Amazon RDS and Aurora.To make any changes to the core of PostgreSQL, all of the ramifications of those changes have to be thought out in advance. Fortunately, we have a whole community of people (some of them proudly part of our own organization) that are really, really good at thinking things out in advance.Query to show your current transaction ages:with overridden_tables as (
  select
    pc.oid as table_id,
    pn.nspname as scheme_name,
    pc.relname as table_name,
    pc.reloptions as options
  from pg_class pc
  join pg_namespace pn on pn.oid = pc.relnamespace
  where reloptions::text ~ 'autovacuum'
), per_database as (
  select
    coalesce(nullif(n.nspname || '.', 'public.'), '') || c.relname as relation,
    greatest(age(c.relfrozenxid), age(t.relfrozenxid)) as age,
    round(
      (greatest(age(c.relfrozenxid), age(t.relfrozenxid))::numeric *
      100 / (2 * 10^9 - current_setting('vacuum_freeze_min_age')::numeric)::numeric),
      2
    ) as capacity_used,
    c.relfrozenxid as rel_relfrozenxid,
    t.relfrozenxid as toast_relfrozenxid,
    (greatest(age(c.relfrozenxid), age(t.relfrozenxid)) > 1200000000)::int as warning,
    case when ot.table_id is not null then true else false end as overridden_settings
  from pg_class c
  join pg_namespace n on c.relnamespace = n.oid
  left join pg_class t ON c.reltoastrelid = t.oid
  left join overridden_tables ot on ot.table_id = c.oid
  where c.relkind IN ('r', 'm') and not (n.nspname = 'pg_catalog' and c.relname <> 'pg_class')
    and n.nspname <> 'information_schema'
  order by 3 desc)
SELECT *
FROM per_database;Adapted from Postgres-CheckupMany enhancements have already been made to PostgreSQL to mitigate the transaction ID wraparound problem and solve it permanently. Here are the steps on the way to the solution.The PostgreSQL system catalogs have already been enhanced to a 64-bit (eight-byte) transaction ID.The functions and procedures of PostgreSQL have been expanded to 64-bit transaction ID parameters and outputs.The backends (query worker processes) can deal with 64-bit transaction IDs.Work has been done on the utilities of PostgreSQL (such aspg_basebackup) that previously assumed 32-bit integer transactions.Replication,VACUUM, and other processes have been enhanced for 64-bit transactions.A lot of other ""stuff."" Many smaller incidental fixes that were based on 32-bit assumptions needed modification.The goal of all of these changes is to eventually move to a 64-bit transaction counter for the entire system.Where do we go from here?There's a bit of bad news. I'm going to close my eyes while I write this, so I won't have to look at your face while you read it.Updating the user tables in your database to use 64-bit transaction counters will require rewriting all of your data. Remember at the beginning, where I said the transaction counter was a per-row solution? Oh, yeah.That means that its limitations are also per row. There are only eight bytes reserved forxminand eight bytes forxmaxin the row header. So, every single row of data in the database is affected.At some point, there will be a major version of PostgreSQL that requires a data dump, replication,pg_upgradeor another such process to re-create every row in the database in the new format. It is true that every major version of PostgreSQLcouldchange the format of data on disk.Thepg_upgradeutility will not be able to use symlinks or hardlinks for the upgrade. These links usually allow for some efficiency while upgrading. There will be no such shortcuts when the ""fix"" for transaction ID wraparound is put into place.Okay, now for the good news. We will all be in retirement (if not taking a dirt nap) when the next bunch ofsuckersengineers has to deal with this issue again. 263is not double the number of transactions. It is 9,223,372,034,707,292,160 (nine quintillion) more.What to do while you're waiting for infinityYou can still make use of some basic mitigation strategies for transaction ID wraparound failures:Make the autovacuum process more aggressive to keep up with maintaining the database.Use custom settings to make the autovacuum process more aggressive for the most active tables.Schedule vacuumdbto do additional vacuuming tasks for PostgreSQL to catch up faster.Vacuum theTOASTtables separately so the autovacuum has a better chance of catching up.REINDEX CONCURRENTLYmore frequently so that the autovacuum has less work to do.CLUSTER ON INDEXwill re-order the data in the table to the same order as an index, thus ""vacuuming"" the table along the way.VACUUM FULL, which blocks updates while vacuuming but will finish without interruption. Let me say that again. There will be no writes whileVACUUM FULLis running, and you can't interrupt it. 😠Switch over to a secondary. The transaction counter will be reset to one when the system is restarted. (There are no transactions in flight, are there? 😄)Use batching forINSERT,UPDATE, andDELETEoperations.  The counter is issued per transaction (not row), so grouping operations helps reserve counters.All of these strategies are basically the same thing. The objective is to ensure the tail number (oldest transaction) moves forward as quickly as possible. This will prevent you from ending up in a ""transaction ID wraparound"" scenario.🙂 ♥️ 👍Documentation and ResourcesCheck out the PostgreSQL documentation onroutine vacuumingto prevent transaction ID wraparound failures.The Timescale Docs alsotroubleshoot transaction ID wraparound exhaustion.How Timescale Can HelpWhile Timescale—also built on the rock-solid foundation of PostgreSQL— does not solve transaction ID wraparound failure, it can help you prevent it since our ingestion inherently batches the data by design after youinstalltimescaledb-parallel-copy.Of course, you can do this for yourself with transaction blocks, but our tools will do the right thing automatically.We also provide ageneral-purpose job schedulerthat can be useful for addingVACUUMandCLUSTERoperations.So, if you want to mitigate the chances of ever dealing with XID wraparound problems while enjoyingsuperior query performance and storage savings compared to vanilla PostgreSQLorAmazon RDS for PostgreSQL, try Timescale.Sign up now(30-day free trial, no credit card required) for fast performance, seamless user experience, and the best compression ratios.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-fix-transaction-id-wraparound/
2023-03-07T14:30:08.000Z,A PostgreSQL Developer’s Perspective: Five Interesting Patches From January’s Commitfest,"🐘The PostgreSQL community organizes patch reviews into “commitfests” which last for a month at a time, every other month. In this series, our very own PostgreSQL developer advocate and expert, Chris Travers, will discuss a few patches that may be of interest to PostgreSQL users after each commitfest. This is intended to provide a sense of the progress and future of PostgreSQL as a major player in the database world.January’s Commitfesthas completed with 70 patches committed, 183 deferred to the next commitfest, 14 withdrawn, and 12 returned with feedback.As always, there are a large number of interesting patches, and I have selected five patches to discuss today,one of which was reviewed last time. Three of these patches have been committed, one was withdrawn, and the last one was deferred to the next commitfest.In most cases, I avoid selecting patches where I am a reviewer, but I have decided to make an exception here and discuss the 64-bit XID patch. In the interest of full transparency, I am a reviewer of the patch, and my discussion here is not intended to replace or extend my participation in that review process.Committed Improvements in Performance and Ease of UseImprove tab completion forALTER FUNCTION/PROCEDURE/ROUTINEThis is perhaps a trivialimprovement, but it addresses something I find annoying when working on prototyping systems. It also affects some production issues if people need to alter functions, etc., for production impacts.This patch affects psql specifically and applies only to the standard command-line, scripting, and administration tool that ships with PostgreSQL. The psql client is one of the most scriptable database clients I have ever worked with. It can generate output in several formats (including HTML and LaTeX) and has several features specifically for use in scripting tools. Therefore, it’s my go-to tool for any administrative task for working with the database.Why this mattersThis is a small improvement, which eliminates a small annoyance. I am used to using\dffor autocomplete of functions and procedures. This is likely to lead to a faster resolution in some cases.This patch may seem minor, but it represents an example of how PostgreSQL just keeps getting easier to use via the existing command line tools. PostgreSQL keeps getting better and better.postgres_fdw—useTABLESAMPLEwhen analyzing foreign tablesPostgreSQL can access data managed or stored in other ways using “foreign data wrappers.” Foreign data wrappers are pieces of software that plug into PostgreSQL and allow PostgreSQL to access data not stored in PostgreSQL-managed relations.PostgreSQL still needs statistical information about the table's contents to properly plan queries so that they can run as quickly as possible. Current versions of PostgreSQL gather such statistics by sampling rows on the client side, which means that PostgreSQL will query the remote server for all data, then use a subset of it to populate the statistics, discarding the rest.  This makes the PostgreSQL foreign data wrapper less than ideal for large foreign tables.Apatchhas now been committed that uses PostgreSQL’sTABLESAMPLEoption to request a statistical sample of the data. This improves network traffic and processing time. Unfortunately, as I understand the patch, it does not reduce disk I/O on the storing server, but this is still a massive improvement.Why this mattersDistributed data environments, in my experience, become more common as data volumes grow. This means that data volume issues are fairly common in these environments. This patch improves performance and load in a couple of ways.The first obvious issue is that network traffic is reduced. However, filtering is often less expensive close to the storage (as is downsampling). This is why proper use of aggregation often improves database load, even on highly loaded systems. While we don’t have real-world data yet on the impact we will see on real-world workloads, I expect this to be a very important step forward.+infinity for dates and timestampsThis is another small improvement that highlights perpetual improvements in PostgreSQL. A smallpatchhas been proposed and accepted, allowing ‘+infinity’ as a synonym for ‘infinity.’  This works for numeric, date, and timestamp values.I originally selected this patch because my knowledge of the date type was, pardon the pun, out of date. I had missed that, in version 8.4, dates could also be infinite (prior, I had to use timestamps to get infinite values and had missed the change). Although this patch ended up being minor, it turned me to older improvements and a story of continuous improvement of PostgreSQL.All the way back, at least to PostgreSQL 6.3, PostgreSQL has supported special values for timestamps for infinite values. The idea is that you can have an infinity timestamp that is later than all other timestamps and a -infinity timestamp earlier than any other timestamp.These can often be used as sentinel values, for example, indicating an open-ended validity. While this is available for timestamps, only much later (in version 8.4) was similar functionality for the date type. I understand this was done during a series of general improvements and reworkings of the date type itself.Finally, in the next version, we will be able to support more consistent notation for infinite values, allowing easier testing and software development.Why this mattersPostgreSQL is improving even in corners that may seem exotic from the outside. This small change allows easier software development in cases where infinite values are used. It will be noticed.Withdrawn Patch of InterestNew strategies for freezing, advancingrelfrozenxidearlyOne issue that users of large databases face is the dreaded transaction ID wraparound.This problem occurs whenVACUUMfalls behindand cannot advance the counterpoints, which are known to be always in the past. That’s a simplification, but it generally describes the problem.Apatch setwas proposed which was intended to address this problem by providing more efficient management of transaction IDs. Unfortunately, on review, a number of potential problems were discussed, and the patch was withdrawn.This work direction is essential, particularly in combination with the 64-bit XID patch discussed below. I look forward to seeing more innovative approaches to this problem–possibly including storage approaches that might not face the problem at all.It is the mark of an exceptionally professional and talented community that we can put patches through blistering reviews and that people can and do withdraw patches on their own accord when problems are pointed out. This sort of behavior is part of what makes PostgreSQL rock-solid. And withdrawal doesn’t mean another approach or perhaps further iteration won’t be submitted at some point.Patch Moved to Next CommitfestAdd 64-bit XIDs into PostgreSQL 16This is a second, much older patch set that helps to address a different aspect of transaction ID wraparounds. There is some disagreement in the community about why this patch is relevant but a general (though not complete) consensus that this is probably, at least in its outlines, a direction we will need to move as a database engine.Thispatch setoriginated fromPostgres Professionalseveral years ago and is one that that company has perpetually tried to push into PostgreSQL. As a result, it represents a multi-year effort that, while it may have been ahead of its time, is rapidly becoming more relevant and urgently needed. I am proud to report that Timescale is now collaborating on this patch and trying to help get it through the review process.My own view of the work has been somewhat skeptical since transaction ID wraparound problems right now mainly occur due to other things—such as autovacuum—not working as expected, and these malfunctions can have different impacts, too (I have faced transaction wraparound problems myself).Whatever disagreements I have with the authors of the patch, it is important to note that 64-bit XIDs are likely to become increasingly crucial in the coming years. With every release, PostgreSQL becomes more capable, and hardware becomes more powerful.One current source of bottlenecks for high-throughput workloads is concurrent transaction handling.  Eliminating this bottleneck allows more data to be loaded faster, which in turn makes the hard limitations of 32-bit transaction IDs far more painful when they start to run out.As a result, transaction ID wraparound problems develop more quickly and can become rapidly more severe, and experimental storage engines that eliminate the need for vacuuming are likely to add even more urgency as some bottlenecks may be eliminated, and transaction IDs might be used up at ever faster rates. While a few people insist that the only proper answer is to fix vacuuming, the fact is that an eventual shift to 64-bit transaction IDs would allow for workloads that aren’t currently possible.I believe that the work of the community here needs to be to get this patch into an optimal shape (or at least into a shape where, despite the extra storage used to store transaction IDs, it is an improvement in every current workload) and committed. I hope this happens soon.Concluding Thoughts: A Professional Community Improving a Rock-Solid DatabaseA major theme in this patch selection is an attention to detail in potential and actual improvements and a willingness to look past ego or pride in order to do what’s best for the community of this truly worldwide project.Developers may spot small improvements that lead to a much better user experience and submit these changes. Others may try to fix long-standing problems only to eventually realize that the proposed solutions cause more problems and gracefully withdraw their suggestions. Even slow-moving patches can be carried forward by many people working together. This is why PostgreSQL continues to improve so much each time a major version is released.As PostgreSQL evolves, you can add even more functionality to this beloved database.Explore TimescaleDB—it extends PostgreSQLwith features like automatic time-based partitioning and indexing, continuous aggregations, columnar compression, and time-series functionality.And if you’re using a managed service for PostgreSQL,try Timescale—it’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/a-postgresql-developers-perspective-five-interesting-patches-from-januarys-commitfest/
2023-01-19T16:28:52.000Z,The PostgreSQL Job Scheduler You Always Wanted (Use it With Caution),"As a PostgreSQL guy, it really makes you wonder why a built-in job scheduler is not a part of the core PostgreSQL project. It is one of the most requested features in the history of ever. Yet, somehow, it just isn’t there.Essentially, a job scheduler is a process that kicks off in-database functions and procedures at specified times and runs them independently of user sessions. The benefits of having a scheduler built into the database are obvious: no dependencies, no inherent security leaks, fits in your existing high availability plan, and takes part in your data recovery plan, too.ThePostgreSQL Global Development Grouphas been debating for years about including a built-in job scheduler. Even after the addition of background processes that would support the feature (all the way back in 9.6), background job scheduling is unfortunately not a part of core PostgreSQL.So being the PostgreSQL lovers we are atTimescale,we decided to build such a schedulerso that our users and customers can benefit from a job scheduler in PostgreSQL. In TimescaleDB 2.9.1, we extended it to allow you to schedule jobs with flexible intervals andprovide you with better visibility of error logs.The flexible intervals enable you to determine whether the next run of the job occurs based on the scheduled clock time or the end of the last job run. And by “better visibility” of the job logs, we mean that they are also being logged to a table where they can be queried internally. These were extended to prevent overlapping job executions, provide predictable job timing, and provide better forensics.We extensively use the advantage of this internal scheduler for our core features, enabling us to defercompression,data retention, and refreshing of continuous aggregates to a background process (among other things).📝 Editor's note:Learn more about how TimescaleDB's hypertables enable all these features above as a PostgreSQL extension, plus other awesome things like automatic partitioning.This scheduler makes Timescale much more responsive to the caller and results in more efficient processing of these tasks. For our own benefit, the job scheduler needs to be internal to the database. It also needs to be efficient, controllable, and scale with the installation.We made all this power available to you as a PostgreSQL end user. If you're running PostgreSQL in your own hardware, you caninstall the TimescaleDB extension. If you're running in AWS,you can try our platform for free.The PostgreSQL Job Scheduler DebateBut not so fast. Before you start rejoicing, let’s review the reasons that the PostgreSQL Global Development Group chose not to include a scheduler in the database—there'll be educational for you as a word of caution.Rather than rehashing the discussion list on the subject, let's summarize the obstacles that came up in themailing list:PostgreSQL is multi-process, not multi-thread.This simple fact makes having a one-to-one relationship of processes to user-defined tasks a fairly heavy implementation issue. Under normal circumstances, PostgreSQL expects to lay a process onto a CPU (affinity), load the memory through the closest non-uniform memory access (NUMA) controller, and do some fairly heavy data processing.This works great when the expectation is that the process will be very busy the majority of the time. Schedulers do not work like that. They sit around with some cheap threads waiting to do something for the majority of the life of the thread. Just the context switching alone would make using a full-blown process very expensive.Background workers' processes are a relatively small pool by design.This has a lot to do with the previous paragraph, but also that each process allocates the prescribed memory at startup. So, these processes compete with SQL query workers for CPU and memory. And the background processes have priority over both resources since they are allocated at system startup.The next issue is more semantic.There are quite a few external schedulers available. Each one of them has a different implementation of the time management system. That is, there is a question about just how exactly the job should be invoked. Should it be invoked again if it is still running from the last time? Should the job be started again based on clock time or relative to the previous job run? From the beginning or the end of the last run?There are quite a few more questions of this nature, but you get the idea. No matter how the community answers these questions, somebody will complain that the implementation is the wrong answer because\<insert silly mathematician answer here\>.Why We Still Need a PostgreSQL Job SchedulerTimescale doesn't have the luxury of debating how many angels can dance on the head of a pin. As a database service working with large volumes of data in PostgreSQL, we face a hard requirement of background maintenance for the actions of archival, compression, and general storage. Timescale's core features, excludinghyperfunctions, depend on the job scheduler.But, rather than create a bespoke scheduler for our own purposes we built a general-purpose scheduler with a public application programming interface.This general-purpose scheduler is generally available as part of TimescaleDB. You may use it to set a schedule for anything you can express as a procedure or function. In PostgreSQL, that's a huge advantage because you have the full power of the PostgreSQL extension system at your disposal. This list includes plug-in languages, which allow you to do anything the operating system can do.Timescale assumes that the developer/administrator is a sane and reasonable person who can deal with a balance of complexity. That is longhand for ""we trust you to do the right thing.""With Great Power Comes Great ResponsibilitySo, let's talk first about a few best design practices for using the Timescale (PostgreSQL) built-in job scheduler.Keep it short.The dwell time of the background process can lead to high concurrency.  You are also using a process shared by other system tasks such as sorting, sequential scans, and other system tasks.Keep it unlocked.Try to minimize the number of exclusive locks you create while doing your process.Keep it down.The processes that you are using are shared by the system, and you are competing for resources with SQL query worker processes. Keep that in mind before you kick off hundreds or thousands of scheduled jobs.Now, assuming we are using the product fairly and judiciously, we can move on to the features and benefits of having an internal scheduler.Built-In PostgreSQL Job Scheduler: All the Nice StuffNow that we've covered the things that demand caution, here's a list of some of the benefits of using this scheduler:Physical streaming replication will also replicate the job schedule. When you go to switch over to your replica, everything will already be there.You don't need a separate high-availability plan for your scheduler. If the system is alive, so are your scheduled jobs.The jobs can report on their own success or failure to internal tables and the PostgreSQL log file.The jobs can do administrative functions like dropping tables and changing table structure by monitoring the existing needs and structures.When you install Timescale, it's already there.📝Editor's note: Quick reminder that you caninstall the TimescaleDB extensionif you're running your own PostgreSQL database, orsign up for the Timescale platform(free for 30 days).How The Job Scheduler WorksThere isa quick introductory article in the Timescale documentation. Click that link if you want more detailed information.The TL;DR version is that you make a PostgreSQL function or procedure and then call theadd_job()function to schedule it. Of course, you can remove it from the schedule using… Wait for it...delete_job().That's it. Really. All that power is at your fingertips, and all you need to know is two function signatures.Something to be aware of while you're using the scheduler is that the job may be scheduled to repeat from the end of the last run or from the scheduled clock time (in TimescaleDB 2.9.1 and beyond). This allows you to ensure that the previous job has completed (by picking from the end of the run) or that the job executes at a prescribed time (making job completion your responsibility).If you feel a bit homesick and just want to look at your adorable job, there's also:SELECT * FROM timescaledb_information.jobs;And, of course, for completeness, there's alwaysalter_job()for rescheduling, renaming, etc.Once your job has been created, it becomes the responsibility of the job scheduler to invoke it at the proper time. The job scheduler is a PostgreSQL background process. It wakes up every 10 seconds and checks to see if any job is scheduled in the near future.If such a job is queued up, it will request another background process from the PostgreSQL master process. The database system will provide one (provided there are any available). The provided process becomes responsible for the execution of your job.This basic operation has some ramifications. We have already mentioned that we need to use these background processes sparingly for resource allocation reasons. Also, there are only a few of them available. The maximum parallel count of background processes is determined bymax_worker_processes. If you need help configuring TimescaleDB background workers,check out our documentation.📝 You can also check out this blog post on tuning TimescaleDB parameters.On my system (Kubuntu 22.04.1, PostgreSQL 14.6), the default is 43. That number is just an example, as the package manager for each distribution of PostgreSQL has discretion about the initial setting. Your mileage **will** vary.Changing this parameter requires a restart, so you will need to make a judgment call about how many concurrent processes you expect to kick off. Add that to this base number and restart your system. Of course, a reasonable number has been added for you in Timescale. Remember the CPU and memory limitations while you are making this adjustment.What to Do With A PostgreSQL Job Scheduler: A Few IdeasThe original reasons for creating this scheduler involve building out-of-the-box features involving data management. That includescompression,continuous aggregates,retention policy implementation,downsampling, andbackfilling.You may want to use this for event notifications, sending an email, clustered index maintenance, partition creation, pruning, archiving, refreshing materialized views, or summarizing data somewhere to avoid the need for triggers. These are just a few of the obvious ideas that jump into my consciousness. You can literally do anything that the operating system allows.WhatNotto DoThis would be a bad place to gum up the locking tables. That is, be sure that whatever you do here is done in a concurrent manner.REFRESH INDEX CONCURRENTLYis better thanDROP/CREATE INDEX.REFRESH MATERIALIZED VIEW CONCURRENTLYis better thanREFRESH MATERIALIZED VIEW. You get it. UseCONCURRENTLY, or design concurrently. Better yet, do things in a tiny atomic way that takes little time anyway.Long-running transactions that create a lot of locks will interfere with the background writer, the planner, and the vacuum processes. If you crank up too many concurrent processes, you may also run out of memory. Please try to schedule everything to run in series. You’ll thank me later.Well Wishes to the Newly Crowned EmperorNow you have the power to do anything your little heart desires in the background of PostgreSQL without having any external dependencies. We hope you feel empowered, awed, and a little bit special. We also hope you will use your new powers for good!Try the Updated Job SchedulerThe job scheduler is available in TimescaleDB 2.9.1 and beyond. If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation. If you are using theTimescale platform, upgrades are automatic, meaning that you already have the scheduler at your fingertips.Keep LearningIf this article has inspired you to keep going with your PostgreSQL hacking,check out our collection of articles on PostgreSQL fine tuning.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/the-postgresql-job-scheduler-you-always-wanted-but-be-careful-what-you-ask-for/
2023-03-09T14:00:05.000Z,Best Practices for Time-Series Data Modeling: Single or Multiple Partitioned Table(s) a.k.a. Hypertables,"Collectingtime-related information, or time-series data, creates massive amounts of data to manage and model. Storing it will require one or more Timescale hypertables, which are very similar to PostgreSQL partitioned tables.Timescale hypertables work like regular PostgreSQL tablesbut offer optimized performance and user experience for time-series data. With hypertables, data is stored in chunks, which work similarly toPostgreSQL’s partitioned tablesbut support multiple dimensions and other features. While we discussed the table layout in theNarrow, Medium, or Wide Table Layoutbest practices article, this time, we addressed whether you should use a single table to store all data versus using multiple tables, as well as their respective pros and cons.Timescale is built upon a relational database model, which means it supports numerous data modeling choices or ways in which data can be organized and laid out. Understanding the database design choices early on is crucial to find the best combination.I started using Timescale long before joining the company, initially using it to store IoT metrics at my own startups. We went through a few different iterations of designs, and migrations between those were everything but fun. Due to that personal experience, one of my biggest goals is to prevent others from suffering through the same.Time-Series Data Modeling: Using our Relational Database ExperienceAs mentioned, Timescale uses a relational data model at its core. Being built on PostgreSQL, we understand we have many ways to store data, including in partitioned tables or just separate ones. A common pattern in the relational world is to divide data into separate tables based on their content, also often referred to asdomainorentity. That means that data belonging to a set of A’s is stored in a different table than a set of B’s, representing differentdomains.That leaves us questioning how the concepts of time-series data and relational domains fit together. Unfortunately, there is no easy answer. Our primary options are “squeezing” all data into a single table, which could have hundreds of columns (basically defining our domain around the idea of “it’s all just metrics”), or splitting data into multiple tables with fewer columns. The latter choice may slice tables in many ways, such as by metric type (temperature is different from humidity, stock symbol A is different from symbol B), customer, data type, and others, or combinations of the previous.Both possibilities have their own set of advantages and disadvantages, which can be split into four commonly seen topics:Ease of useMulti-tenancy / Privacy-related requirements (General Data Protection Regulation or GDPR / California Consumer Privacy Act or CCPA / others)Schema migration or upgrading / Future-proofnessTooling supportI did not choose the above order by accident: the sequential importance of these questions may influence your options further down the line.Pros and ConsAs said before, both design choices have pros and cons, and it’s vital to understand them before making a final data modeling decision. Given the previous set of topics, let’s start by answering a few questions.First and foremost, how important is the ease of use, meaning, are you and your team up to the challenging tasks that need to be solved down the road? Potential “complications” could involve generation patterns for table names or ensuring that similar tables are all upgraded to the same schema level.Next up, are you required to provide a harder level of multi-tenancy, such as storing different customers not just by segregating them using a customer ID but are required to store them in different tables, schemas, or even databases? Is your company bound by regulations (e.g., GDPR or CCPA) where users and customers may have the right to be forgotten? With time-series data being (normally) append-only, removing parts of the data (this specific user’s data) may be tricky.Then we have the question of whether you expect the data schema to change frequently. A large discussion around a future-proof design for hypertables can be found in theNarrow, Medium, or Wide Table Modelbest practices write-up. However, the higher the number of tables, the more they need to be upgraded or migrated in the future, adding additional complexity.Finally, how important is support by additional tools and frameworks, such as ORM (Object Relational Mapping) solutions? While I personally don’t think ORM frameworks are a great fit for time-series data (especially when using aggregations), a lot of folks out there make extensive use of them, so talking about them has its merits.Anyhow, now that we answered those questions, let’s dig into the design choices in greater detail.Single Table DesignsStoring all data into a single table may initially feel irresponsible from a relational database point of view. Depending on how I slice my domain model, though, it could be a perfectly valid option. The design choice makes sense if I consider all stored data (metrics, events, IoT data, stock prices, etc.) as a single domain, namely time series.Single tables make a few things super simple. First of all, and probably obvious to most, is querying. Everything is in the same table, and the queries select certain columns and add additional filters or where clauses to select the data. That is as basic as it can get with SQL. That said, querying data is super simple, not just easy.Upgrading the table’s schema is equally simple. Adding or removing columns implies a single command, and all data is upgraded at the same time. If you have multiple similar tables, you may end up in a situation where some tables are upgraded while others are simply forgotten—no real migration window is needed.Single tables can easily support multiple different values, either through multiple columns (wide table layout), a JSONB column that supports a wide range of data values (narrow table layout), or through columns based on a value’s potential data type (medium table layout).Those three options have pros and cons, though.tsdb=> \d
      Column    |           Type           | Collation | Nullable |      Default
—---------------+--------------------------+-----------+----------+-------------------
 created        | timestamp with time zone |           | not null | now()
 point_id       | uuid                     |           | not null | gen_random_uuid()
 device_id      | uuid                     |           | not null |
 temp           | double precision         |           |          |
 hum            | double precision         |           |          |
 co2            | integer                  |           |          |
 wind_speed     | integer                  |           |          |
 wind_direction | integer                  |           |          |

|                  created | point_id | device_id | temp |  hum | co2 |
 wind_speed | wind_direction |
| 2022-01-01 00:00:00.0+00 |      123 |        10 | 24.7 | 57.1 | 271 |
       NULL |           NULL |Last but not least, single tables play very nicely with tools like ORM frameworks. It is easy to connect a specific set of ORM entities to the hypertable, representing either a full record or the result of an aggregation (which may need a native query instead of an automatically generated one).But as with everything in this world, this choice has a large downside. Since time-series data is designed around the idea of being append-only (meaning that mutating existing records only happens occasionally), it is hard to delete data. Deleting data based on specific requirements is even harder, such as a user or customer asking to have all their data deleted.In that situation, we’d have to crawl our way through potentially years of data, removing records from all over the place. That’s not only a burden on the WAL (Write-Ahead Log) to keep track of the changes, but it also creates loads and loads of I/O, reading and writing.The same is true if we try to store collected and calculated sets of data in the same table. With many systems often having to backfill data (for example, from devices that lost their internet connection for a while and were collecting data locally), calculated values may have to be recalculated. That means that the already stored data must be invalidated (which may mean deleted) and reinserted.Finally, if your company provides different tiers of data retention, good luck implementing this on a single table. It is the previous two issues, but on a constant, more than ugly, basis.Multiple Table DesignsNow that we know about the pros and cons of single table design, what are the differences when we aim for multiple table designs instead?While querying is still simple, querying multiple sets of data simultaneously may be slightly more complicated, involvingJOINsandUNIONsto merge data from the different tables. Requesting multiple sets of data at the same time is often done for efficiency reasons, requiring fewer database round trips and minimizing response time. Apart from that, there isn’t a massive difference in ease of use, except for table names, but we’ll come back to that in a second.One of the major benefits of having multiple tables, especially when sliced by the customer, user, or whatever meaningful multi-tenancy divider for your use case, is the option to quickly react to GDPR- or CCPA-related requests to destroy and remove any customer-related data. In this case, it is as easy as finding all the client’s tables and dropping them. Removing them from backups is a different story, though. 😌The same is true with calculated and collected data. Separating those tables makes it much easier to throw away and recalculate all or parts of the data when late information arrives.Also similar is the previously mentioned data retention. Many companies storing huge amounts of data on behalf of their customers provide different data retention policies based on how much the customer is willing to pay. Having tables sliced by customers makes it easy to set customer-specific retention policies and even change them when a customer upgrades to a higher tier. If this is something you need, themulti-table designis it.However, just as with single tables, multiples have drawbacks too.Besides the already slightly more complicated elements around querying, which are not necessarily a disadvantage, having many tables requires planning a table name schema. The more dimensions we bring into the game (by customer, metric type, etc.), the more complicated our naming schema needs to be. That said, we may end up with table names such as<<customer_name>>__<<metric_type>>. While this doesn’t sound too bad, it can get ugly fast. We’ve all been there before. 😅tsdb=> \dt *.cust_*
                    List of relations
 Schema |            Name            | Type  |   Owner
--------+----------------------------+-------+-----------
 public | cust_mycompany_co2         | table | tsdbadmin
 public | cust_mycompany_humidity    | table | tsdbadmin
 public | cust_mycompany_temperature | table | tsdbadmin
 public | cust_timescale_co2         | table | tsdbadmin
 public | cust_timescale_humidity    | table | tsdbadmin
 public | cust_timescale_temperature | table | tsdbadmin
(6 rows)Tools, such as an ORM framework, may make things even more complicated. Many of those tools are not designed to support arbitrary, runtime-generated table names, making it very complicated to integrate those with this concept. Using different PostgreSQL database schemas per customer and lowering the number of dimensions may help.There is one more complexity: upgrading and migrating tables. Due to the multiple table design, we may end up with many similar tables segregated by the additional dimensions chosen. When upgrading the table schema, we need to ensure that all those tables eventually end up in a consistent state.However, many automatic schema migration tools do not easily support that kind of multi-table migration. This forces us to fall back on writing migration scripts, trying to find all tables matching a specific naming schema, and ensuring that all are upgraded the same way. If we miss a table, we’ll figure it out eventually, but probably when it’s too late.The TL;DRNow that you’ve laid it all out and answered these questions, you can look at the requirements and see where your use case fits.Some hard requirements may make your choice obvious, while some “nice-to-have” elements may still influence the final decision and hint at what may become a harder requirement in the near or far future.Single Table DesignMultiple Table DesignEase of UseEasySomewhat easyMulti-Tenancy /Privacy RegulationsHardEasyFuture-ProofnessEasySomewhat hardTooling SupportEasyHardWhile thesingle table designis very easy to get started with, it may be a hard sell if you need to comply with regulations. However, themultiple table designis certainly more complicated to manage and use correctly.So What's the Best One for Me?There’s never a one-size-fits-all answer, or as consultants love to put it: it depends.Unlike the design choices around the table layout, it’s too complex to make a real recommendation. You can only try to follow the suggested process of answering the questions above and looking at the answers to see which points represent hard requirements, which could become hard requirements, and which are simply nice to have. That way, you’ll likely find the answer to match your use case.Also, remember that you may have different use cases with diverging requirements, so one use case may end up being perfectly fine running as a single table design, while the other one(s) may need multiple tables.Plus, you have the chance to mix and match the benefits of both solutions. It was kind of hinted at in the text already, but it is possible to use a simplified multiple table design (for example, per metric type) and separate the customer dimension into a PostgreSQL database schema, with each schema representing one customer.Similarly, it is possible to use the schema to separate customers and store all metrics/events/data for that specific customer in a single hypertable. There are plenty of options, only limited by your imagination.Whatever you end up with, try to be as future-proof as possible. Try to imagine what the future will hold, and if you’re unsure whether a somewhat harder requirement will become a must-have, it may be worth considering it as a hard requirement now just to be on the safe side.If you want to start designing your hypertable database schema as soon as possible, ensuring you get the best performance and user experience for your time-series data while achieving incredible compression ratios, check outTimescale.If you’re looking to test it locally or run it on-prem, then we’ve got you covered: have alook at our documentation.Learn moreBest Practices for (Time-)Series Metadata TablesBest Practices for Time-Series Data Modeling: Narrow, Medium or Wide Table LayoutIngest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/best-practices-for-time-series-data-modeling-single-or-multiple-partitioned-table-s-a-k-a-hypertables/
2023-01-11T14:30:21.000Z,A PostgreSQL Developer's Perspective: Six Interesting Patches From November's Commitfest,"🐘The PostgreSQL community organizes patch reviews into “commitfests” which last for a month at a time, every other month. In this series, our very own PostgreSQL developer advocate and expert, Chris Travers, will discuss a few patches that may be of interest to PostgreSQL users after each commitfest. This is intended to provide a sense of the progress and future of PostgreSQL as a major player in the database world.November’s commitfestis over with 94 committed patches, 24 patches returned with feedback, 172 patches moved to the next commitfest, 4 rejected, and 13 withdrawn.From a PostgreSQL developer’s perspective and beyond, there was a wide variety of improvements proposed, and while most of the committed patches appear less significant than those of the previous commitfest, a lot of work was actually done. Not every commitfest sees blockbuster patches committed, but there is often considerable progress in the review stage, which can be seen in future releases.In this blog post, I have selected six patches, two committed, and four moved to the next commitfest. I did not select patches I am a reviewer for and focused on other patches I think are important for a wide range of users.PostgreSQL Developer POV: Two Exciting Security Improvements CommittedBoth committed patches I selected are important security improvements that make certain scenarios far more manageable than before.Allowing the use of regular expressions for usernames inpg_hba.confThe firstpatchallows regular expressions in the username field for thepg_hba.conf. While this feature likely has many caveats, it solves some crucial problems in specific environments.Why this mattersNormally, the preferred approach to group management with thepg_hba.confis to use role membership for selectors—this is still likely to be the primary method in most cases going forward. However, some specific situations can be significantly simplified by this patch, particularly trust-basedKerberosauthentication from different domains (which we discussed in a previous post of this series).By default, users connecting to PostgreSQL and authenticating via Kerberos use the Kerberos principal name: PRINCIPAL@DOMAIN, where PRINCIPAL is the name of the Kerberos account, and DOMAIN is the Kerberos domain name. These can look like email addresses.In current versions of PostgreSQL, users do not have assumed access to database objects. These are usually assigned to roles that users are then granted use for. Access can be given to the public as well, which is the same as granting to all current and future roles.These roles can also be used to grant the right to connect to the database via Kerberos or other authentication technologies in thepg_hba.conf, which uses ordered rules to assign and restrict access to the database. For complex security environments, it is often vital to use thepg_hba.confand role-based access controls as separate, related layers in a system defending against unauthorized access. This is particularly important where external partners or departments might be granted access to only one database on a server.This patch then enables the use of rules like “allow the sales department to access the sales database but no others” while still requiring different roles to access anything in that sales database.This is another crucial step in further improving Kerberos (and henceActive Directory Integration) for PostgreSQL.Allow grant and revoke of vacuum and analyze privileges on tablesCurrently, PostgreSQL requires special privileges to vacuum and analyze tables. Database or table owners are able to do these, as can superusers, but apatchhas been committed that allows this to be delegated to other roles and users.Why this mattersMany higher-volume databases require vacuuming on a schedule because autovacuum can become overwhelmed and bogged down. Some database administrators also prefer cost-based delays for autovacuum operations while not having them for manual vacuuming.Currently, the scripts tend to have to run either as a database superuser or as a database or table owner. This is a very strong limitation, and it means that scripts and users that can vacuum tables can do far more than they need to, including destructive operations such as dropping tables.Having this particular privilege available for delegation means that such scripts can be limited to only vacuuming and analyzing tables. This improves security in a critical database operation and reduces the opportunity for things to go wrong.Data Encryption Patches in ReviewEfforts to bring transparent data encryption to PostgreSQL in various forms are proceeding.  This tremendous initiative is unlikely to be committed in a single run. Instead, it will likely be a perpetual topic with many smaller patches which may be committed one at a time.Transparent data encryption has many critical pitfalls that have to be addressed. These patches are likely to go through extensive, painstaking review before being accepted. That process may be long and drawn out.Transparent column encryptionPeter Eisentrauthas proposed apatchfor transparent column encryption. This is a work in progress and not directly part of the Transparent Data Encryption effort, but I am covering it here due to it being closely related and having many of the same problems to address, in particular key management problems.The approach of this patch is to provide key management capabilities for client-side encryption. Column encryption keys are encrypted with column master keys. The database does not store the master keys, just the encryption keys, though they are encrypted with the master keys. This allows key management in a sane way. The client is then responsible for encrypting and decrypting the data.Why this mattersUnlike transparent data encryption, which occurs on the page or block level and encrypts entire tables, encrypting particular columns can be used to protect very specific data against both unauthorized offline retrieval and bulk access.For example, Payment Card Industry Data Security Standards (PCI-DSS) require not merely that credit card data is encrypted but also that someone cannot look at a list of credit card numbers (the data has to be accessed one record at a time).  When both the client application and the storage use these technologies, it can be assured that a database administrator will not easily be able to pull a list of credit card numbers directly.This sort of feature, if it moves forward, would help PostgreSQL compete in high-security PCI-DSS and other related environments.This patch is not likely to get in right away, but it may become the foundation for more high-security features for PostgreSQL in the coming years.Transparent Data Encryption key management patchesDavid Christiansenhas taken on the task of getting Transparent Data Encryption into PostgreSQL. This is a large patch set and has been broken down into a significant number of patches.One of this endeavor's hardest and most critical areas is the key management problems. Onepatch setaddresses this problem.Why this mattersTransparent Data Encryption guarantees that you cannot access the files for a database without an externally managed encryption key. This ensures that filesystem-level or device-level storage cannot be stolen and then easily accessed.Transparent Data Encryption is well-supported on some other relational database systems, but one hard part here is key management and ensuring that compromising information is never logged. Keys must never be logged, nor can information needed to access keys. Getting this part right is very important.This set of patches seems to be moving forward. I believe Crunchy Data’s commitment to this effort is rock solid, and one way or another, we will get top-notch Transparent Data Encryption in PostgreSQL.Two More Patches64-bit XIDs Simple Least Recently Used patchPostgreSQL uses aSimple Least Recently Used (SLRU) cachefor transaction status information.  Internally, transaction IDs (XIDs) are 64-bit integers in some places, but on disk are a 32-bit integer used both in tuple headers forxminandxmaxfields and an offset for the files in thepg_xactdirectory for on-disk lookups for transaction status.  The SLRU cache uses 32-bit integers internally.A newpatchmoves the SLRU in-memory format from 32-bit to 64-bit format.Why this mattersThere is slow progress toward getting all transaction IDs moved to a 64-bit format across the board. This is important because PostgreSQL and hardware are both becoming more capable, and as a result, the risks of transaction ID wraparound-related outages are slowly increasing. A move to 64-bit transaction ID is not a magic bullet for these problems as underlying causes of the wraparound—this represents an important step to this effect.Another vital aspect of this patch is that it removes a critical 32-bit/64-bit transaction ID conversion point. This provides a long-term reduction in the potential footprint of this subsystem as a source of serious bugs, including those that can result in data loss.This patch is awaiting review.Robert Haasand others have expressed a strong interest in it. I believe it will move forward.Logical replication of Data Definition LanguageOne major challenge with logical replication is that it only applies toINSERT,UPDATE,DELETE, andTRUNCATEcommands and that these have problems with changes to database schemas. This is one major issue and requires careful handling and coordination of database schema changes across systems. Apatchhas been submitted and is awaiting changes that replicate Data Definition Language (DDL) as well.Why this mattersLogical replication poses a significant number of challenges for managing database schemas.  Sometimes this can result in subscribers and publishers even being run by different parts of an organization. Human coordination over such changes is tedious and complex and can result in stale replicated data or operational problems.Additionally, running the same DDL statements on the replica is not necessarily safe. If a DDL statement references any volatile constant (such as‘today’::date) or function (such asnow()), there is no guarantee that the replicated table will have the same constraints. This can also break logical replication.Concluding Thoughts: Interesting Incremental ChangesThe November commitfest provided a number of small yet interesting changes and progress on larger projects and patch sets. The incremental security improvements mentioned are exciting and will help make PostgreSQL more competitive in many environments. Additionally, we see slow but solid progress on several larger projects.As PostgreSQL continues to evolve, you can add even more functionality to this beloved database.Explore TimescaleDB—it extends PostgreSQLwith features like automatic time-based partitioning and indexing, continuous aggregations, columnar compression, and time-series functionality.And if you’re using a managed service for PostgreSQL,try Timescale—it’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/a-postgresql-developers-perspective-six-interesting-patches-from-novembers-commitfest/
2023-09-14T13:26:12.000Z,Advice on PgBouncer From a Support Engineer,"Are you looking for an easy way to increase the number of client connections you can establish to your Timescale database? Would you like to reduce the compute (CPU/RAM) overhead of managing (starting-stopping-maintaining) all of those connections?If you’re nodding your head as you read these questions, you may want to consider enabling connection pooling viaPgBouncerfor your service—or perhaps you’re using it already.  As a support engineer at Timescale, I assist our customers daily with their database configuration.This includes connection pooling, specifically PgBouncer, which is the base of our own connection pooler. So I’ve learned one thing or two about it!via GIPHYIn this blog post, I’ll lay out some tips on implementing pgBouncer while avoiding some common mistakes we’ve seen among our customers.🎉You can now boost your application's (and Postgres') performance with prepared statements and PgBouncer's transaction mode.Quick Introduction: PgBouncer in TimescaleAt Timescale, we built our implementation of connection pooling onPgBouncer, a self-described “lightweight connection pooler for PostgreSQL,” since it is a widely used tool by many in our community and performs its function well.To start using PgBouncer in Timescale,you first need to enable the feature in the web console. Once you’ve done this, you'll be provided an additional service URI with a specified port (for PgBouncer) and will be able to connect to either the ""session pool"" (named ""tsdb"") or the ""transaction pool"" (called ""tsdb_transaction""). You will also still be able to make direct connections (via the original service URI and port) to your database server when or if you need to.PgBouncer maintains up to a specified number of dedicated ""server"" connections to PostgreSQL for each pool, as a fluctuating number of clients comes and goes, gets a connection from the pool, runs its queries, and gathers results.When the client disconnects from the connection pool, instead of terminating and deallocating the server connection (which can be very expensive in terms of compute utilization when it happens frequently), it is returned to the pool to be used by another client.On a sufficiently busy system, you can see a substantial benefit of PgBouncer performing this work, along with helping to manage traffic to avoid problems on your database server and free up its resources for query and storage operations.You'll likely see the greatest improvement (reduction) in resource utilization if you are able to take full advantage of the transaction pool (tsdb_transaction). In transaction mode, server connections are added back to the pool after each transaction is completed or rolled back. Almost immediately made available to another client that may be waiting, this allows you to serve a very large number of clients with a relatively small number of dedicated server connections. 💪Practical Tips for PgBouncer SuccessNow, let’s get into the advice. PgBouncer is an awesome tool that will give you so many benefits, but like all powerful things, it must be used with great responsibility. There are some common implementation mistakes that we see people making often—you can avoid them by following these tips:Always monitor your Postgres connection poolFirst things first: when setting up a connection pool, it is highly recommended to monitor behavior.To do so, you can access the PgBouncer administrative console and just specify ""pgbouncer"" as the pool (or database) name, like this:psql postgres://tsdbadmin@HOST:PORT/pgbouncer?sslmode=requireThen, you can run many of the (read-only)SHOWcommands:pgbouncer=# show help;
NOTICE:  Console usage
DETAIL:
        SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|USERS|VERSION
        SHOW PEERS|PEER_POOLS
        SHOW FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEM|STATE
        SHOW DNS_HOSTS|DNS_ZONES
        SHOW STATS|STATS_TOTALS|STATS_AVERAGES|TOTALSIt can be helpful to run some of these commands interactively to get a better understanding of how the pools behave under current conditions. For example, you can see how many clients and servers are active, waiting, or being canceled with theSHOW POOLScommand:pgbouncer> show pools;

 	database 	 |   user	  | cl_active | cl_waiting |
------------------+-----------+-----------+------------+
 pgbouncer    	 | pgbouncer |     	1  |      	   0 |
 postgres     	 | postgres  |     	0  |      	   0 |
 tsdb         	 | tsdbadmin |     	2  |      	   0 |
 tsdb_transaction| tsdbadmin |    	11 |      	   0 |Finally, here is a simple bash script you can use to periodically (default every 90 seconds) collect CSV results of thoseSHOWcommands up to a specified (default 20 minutes) time. This is a rudimentary—but effective—tool to observe some of those statistics change over a period of time.#!/usr/bin/env bash
export PGBOUNCER_URL=""${PGBOUNCER_URL:-empty}""
export EXEC_INTERVAL=""${EXEC_INTERVAL:-90}""
export END_AFTER_MIN=""${END_AFTER_MIN:-20}""

echo ""gathering pgbouncer metrics""
echo ""pgbouncer url: $PGBOUNCER_URL""
echo ""run every $EXEC_INTERVAL seconds""
echo ""end after $END_AFTER_MIN minutes""

declare -a arr=(databases pools clients servers users sockets active_sockets lists dns_hosts dns_zones stats stats_totals stats_averages stats_totals)

if [ $PGBOUNCER_URL != ""empty"" ]; then
	psql --csv $PGBOUNCER_URL -c ""show config;"" | sed ""s/$/,dt`date +%Y%m%d%H%M%S`/"" > ./pb_config.csv

	for key in ""${arr[@]}""; do
		psql --csv $PGBOUNCER_URL -c ""show ${key}"" | sed ""s/$/,at_date_time/"" | head -n1 > ./pb_${key}.csv
	done

	touch .get_pb_metrics;
	while [ -f "".get_pb_metrics"" ]; do
		for key in ""${arr[@]}""; do
			psql --csv -t $PGBOUNCER_URL -c ""show ${key}"" | sed ""s/$/,`date +%Y%m%d%H%M%S`/"" >> ./pb_${key}.csv
		done
		sleep $EXEC_INTERVAL
		find .get_pb_metrics -mmin +$END_AFTER_MIN -type f -exec rm -fv {} \;
	done

	tar czvf ./pb_metrics.tar.gz ./pb*.csv
	rm ./pb*.csv
else
	echo ""you must set PGBOUNCER_URL""
fiYou can save that to a file (make it executable), set a properPGBOUNCER_URLenvironment variable, run it, and watch the tail of your favorite CSV. When it's done, collect them into tables for query if you like.I know, I know, this all sounds awesome (because it is). But before you change all of your connection strings, keep reading for more best practices to ensure you get the absolute most out of this feature.Avoid session-based features when using the transaction poolOne of the mistakes we often see people making with PgBouncer is using session-based features (such as prepared statements, temporary tables, and SET commands) that can fail or produce unexpected results for customers using the transaction pool.ThePgBouncer FAQ briefly discusses thisand then links to how to disable this feature in JDBC and PHP/PDO.Tip:You may have to configure your framework/library to prevent the use of prepared statements.You will find that some tools, frameworks, and some ""thick"" client software may expect or require a connection to the database in ""session"" mode. For example, if you pointpg_activityto your transaction pool, it may run for a short while, but you'll probably see errors similar to this:prepared statement ""_pg3_9"" does not existSimilarly,pgcliseems to work fine for most things, but if you use the\watchcommand, you may see something such as:prepared statement ""_pg3_0"" already existsFor another example to demonstrate why you should avoid session-based features while using the transaction pool, connect psql and do this:home:/> psql postgres://tsdbadmin@HOST:PORT/tsdb_transaction?sslmode=require

tsdb_transaction> show search_path;
+-----------------+
| search_path     |
|-----------------|
| ""$user"", public |
+-----------------+

tsdb_transaction> set search_path = ""$user"", custom, public;

tsdb_transaction> show search_path;
+-------------------------+
| search_path             |
|-------------------------|
| ""$user"", custom, public |
+-------------------------+

tsdb_transaction> select * from temp_table;
relation ""temp_table"" does not exist
LINE 1: select * from temp_table
                      ^

tsdb_transaction> create temporary table temp_table as select 'foo' as bar;

tsdb_transaction> select * from temp_table;
+-----+
| bar |
|-----|
| foo |
+-----+

tsdb_transaction> \q
Goodbye!


home:/> psql postgres://tsdbadmin@HOST:PORT/tsdb_transaction?sslmode=require
...

tsdb_transaction> select * from temp_table;
+-----+
| bar |
|-----|
| foo |
+-----+

tsdb_transaction> show search_path;
+-------------------------+
| search_path             |
|-------------------------|
| ""$user"", custom, public |
+-------------------------+Notice how, in this example, after disconnecting and reconnecting (terminating the ""client"" and creating a new ""client"" connection) to the transaction pool, the previously created temporary table and setting are visible to this connection (which obviously retrieved the same ""server"" connection which had been added back to the pool).Take max advantage of the transaction pool modeOur experience suggests that most users will get the greatest benefit from using connection pooling when you can take greater advantage of the transaction mode pool. If your use case does not require session-based features, and if most of your transactions are very short, we recommend using the transaction pool.Large volumes of short-lived clients, things like event systems, IoT and sensor networks, and microservices architectures can all benefit from transaction pools. Meanwhile, for those “thick clients” (like your long-running sessions in your robust BI Suite), these connections may better be served by the session pool.Our general guidelines would be:If you do not need and can avoid using any session-based features, use the transaction pool. If you do need those features, use the session pool. These two pools should meet most of your needs, but if you do encounter something that prevents you from using either pool available, you can still directly connect to your service.As always,we’re happy to discuss this further, but we hope these simple tips will save you some precious time.Wrap-UpWith connection pooling via PpgBouncer, we aim to provide a simple way for Timescale users to increase the number of database operations, manage system resources more efficiently, and improve database reliability.We expect to keep learning more about the optimal implementation of pgBouncer as we move forward. We’ll make sure to share them with the community—in the meantime, please let us know in our communitySlack,Forum, or reach out forsupportif you have any feedback or run into any obstacles at all. Also, if you have further advice on how to avoid pgBouncer pitfalls, we’d love to hear it!Excelsior!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/using-pgbouncer-to-improve-your-postgresql-database-performance/
2023-04-06T13:30:44.000Z,How to Fix No Partition of Relation Found for Row in Postgres Databases,"ERROR: No Partition of Relation Found for RowThe error messageERROR: no partition of relation {table-name} found for rowis reported by PostgreSQL (and will appear in the console and the log) when a table has been configured with declarative partitioning, and data isINSERTedbefore a child table has been defined with constraints that match the data. This will cause the insert to fail, potentially losing the data which was in flight.You will find this error message in other PostgreSQL-based databases, such as Amazon RDS for PostgreSQL and Amazon Aurora. But it can be avoided in Timescale when you use ourhypertable abstraction. In this blog post, we’ll explain this database error in more detail to learn why.ExplanationLet’s dive deeper into what causes ano partition of relation found for rowerror. When a table is partitioned using PostgreSQL declarative partitioning, it becomes a parent to which multiple child partitions can be attached. Each of these children can handle a specific non-overlapping subset of data. When partitioning by time (the most common use case), each partition would be attached for a particular date range. For example, seven daily partitions could be attached, representing the upcoming week.When inserts are made into the parent table, these are transparently routed to the child table, matching the partitioning criteria. So an insert of a row that referenced tomorrow would be sent automatically to tomorrow’s partition. If this partition doesn’t exist, then there is a problem—there is no logical place to store this data. PostgreSQL will fail theINSERTand reportno partition of relation {table-name} found for row.How to ResolveThere are two ways around this problem, although neither is perfect. Keep reading to see the Timescale approach withhypertablesthat avoids these pitfalls.Partitions can be made ahead of time—perhaps a scheduler could be used to create a month's worth of partitions automatically in advance. This works in theory (as long as that scheduler keeps running!) but will cause locking issues while the partitions are being created. Plus, it doesn’t account for data in the past or the far future.A default partition can also be added that automatically catches all data that doesn’t have a home, but this is problematic, too, as it collects data that needs to eventually be moved into freshly created partitions. As the amount of orphaned data in the default partition grows, it will also slow down query times.Documentation and ResourcesTimescalehypertables work like regular PostgreSQL tablesbut provide a superior user experience when handling time-series data.Need some advice on how to model your time-series data using hypertables? Read our best practices about choosing between anarrow, medium, or wide hypertable layoutand learn when to usesingle or multiple hypertables.How Timescale Can HelpAs mentioned earlier, another solution is enabling the TimescaleDB extension and converting the table into a hypertable instead of using PostgreSQL declarative partitioning. This removes the need to worry about partitions (which in Timescale jargon are called chunks), as they are transparently made when inserts happen with no locking issues.You’ll never have to see this error, worry about scheduling potentially disruptive partition creation, or think about default partitions ever again!New to Timescale?Sign up for Timescale(30-day free trial, no credit card required) for fast performance, seamless user experience, and the best compression ratios.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-fix-no-partition-of-relation-found-for-row/
2023-06-15T14:10:00.000Z,Implementing ASOF Joins in PostgreSQL and Timescale,"What Is anASOFJoin?AnASOF(or ""as of"") join is a type of join operation used when analyzing two sets of time-series data. It essentially matches each record from one table with the nearest—but not necessarily equal—value from another table based on a chosen column. Oracle supports this out of the box using a non-standard SQL syntax, but unfortunately, PostgreSQL does not provide a built-inASOFkeyword.The chosen column needs to have some concept of range for theASOFoperation to work. You may think of it as being the ""closest value,"" but not exceeding the comparison. It works for string (alphabetical), integer (ordinal), float (decimal), and any other data type that has an idea of ORDER. Because timestamps are near and dear to our hearts at Timescale, we will demonstrate with time and date columns.✨Want to understandhow the PostgreSQL parser picks a join method or join types? Check out this article!Performing this operation in PostgreSQL takes a bit of effort. This article aims to delve deeper intoASOF-style joins and how to implement similar functionality in PostgreSQL by subselecting data or other join types.UnderstandingASOFJoinsASOFjoins are a powerful tool when dealing with time-series data. In simple terms, an ASOF join will, for each row in the left table, find a corresponding single row in the right table where the key value is less than or equal to the key in the left table.This is a common operation when dealing with financial data, sensor readings, orother types of time-series data where readings might not align perfectly by timestamp.For a simple example, consider the real-world question, ""What was the temperature yesterday at this time?"" It is very unlikely that a temperature reading was taken yesterday at exactly the millisecond that the question is asked today. What we really want is ""What was the temperature taken yesterday up to today's time stamp?""This simple example becomes a lot more complex when we start comparing temperatures day over day, week over week, etc.ImplementingASOFJoins in TimescaleEven though PostgreSQL does not directly supportASOFjoins, you can achieve similar functionality using a combination of SQL operations. Here's a simplified step-by-step guide:Step 1: Prepare your dataEnsure your data is in the correct format for theASOFjoin. You'll need a timestamp or other monotonically increasing column to use as a key for the join.Suppose you have two tables,bidsandasks, each containing a timestamp column, and you want to join them by instrument and the nearest timestamp.CREATE TABLE bids (
    instrument text,
    ts TIMESTAMPTZ,
    value NUMERIC
);
--
CREATE INDEX bids_instrument_ts_idx ON bids (instrument, ts DESC);
CREATE INDEX bids_ts_idx ON bids (ts);
--
CREATE TABLE asks (
    instrument text,
    ts TIMESTAMPTZ,
    value NUMERIC
);
CREATE INDEX asks_instrument_ts_idx ON asks (instrument, ts DESC);
CREATE INDEX asks_ts_idx ON asks (ts);
--Normally you'd make both these tables into hypertables with thecreate_hypertablefunction (because you're a super educated Timescale user), but in this case, we aren't going to, as we won't be inserting much data (and we also have some Timescale magic to show off 🪄).Step 2: Insert some test dataNext, we'll create data for four instruments,AAA, BBB, NCD,andUSD.INSERT INTO bids (instrument, ts, value)
SELECT 
   -- random 1 of 4 instruments
  (array['AAA', 'BBB', 'NZD', 'USD'])[floor(random() * 4 + 1)], 
   -- timestamp of last month plus some seconds
  now() - interval '1 month' + g.s, 
   -- random value
  random()* 100 +1
FROM (
  -- 2.5M seconds in a month
  SELECT ((random() * 2592000 + 1)::text || ' s')::interval s 
  FROM generate_series(1,3000000)) g;
INSERT INTO asks (instrument, ts, value)
SELECT 
   -- random 1 of 4 instruments
  (array['AAA', 'BBB', 'NZD', 'USD'])[floor(random() * 4 + 1)], 
   -- timestamp of last month plus some seconds
  now() - interval '1 month' + g.s, 
   -- random value
  random()* 100 +1
FROM (
  -- 2.5M seconds in a month
  SELECT ((random() * 2592000 + 1)::text || ' s')::interval s 
  FROM generate_series(1,2000000)) g;Step 3: Query the data using a sub-selectTo mimic the behavior of anASOFjoin, use aSUBSELECTjoin operation along with conditions to match rows based on your criteria. This will run the sub-query once per row returned from the target table. We need to use theDISTINCTclause to limit the number of rows returned to one.This will work in vanilla Postgres, but when we are using Timescale (even though we aren't using hypertables yet), we get the benefits of a Skip Scan, which will supercharge the query (for more information on this check ourdocsorblog post about how Skip Scan can give you an 8,000x speed-up).SELECT bids.ts timebid, bids.value bid,
    (SELECT DISTINCT ON (asks.instrument) value ask
    FROM asks
    WHERE asks.instrument = bids.instrument
    AND asks.ts <= bids.ts
    ORDER BY instrument, ts DESC) ask
FROM bids
WHERE bids.ts > now() - interval '1 week'QUERY PLAN                                                                               
-------------------------------------------------------------------------
 Index Scan using bids_ts_idx on public.bids  
    (cost=0.43..188132.58 rows=62180 width=56) 
    (actual time=0.067..1700.957 rows=57303 loops=1)
   Output: bids.instrument, bids.ts, bids.value, (SubPlan 1)
   Index Cond: (bids.ts > (now() - '7 days'::interval))
   SubPlan 1
     ->  Unique  (cost=0.43..2.71 rows=5 width=24) 
                (actual time=0.027..0.029 rows=1 loops=57303)
           Output: asks.value, asks.instrument, asks.ts
           ->  Custom Scan (SkipScan) on public.asks  
                  (cost=0.43..2.71 rows=5 width=24) 
                  (actual time=0.027..0.027 rows=1 loops=57303)
                 Output: asks.value, asks.instrument, asks.ts
                 ->  Index Scan using asks_instrument_ts_idx on public.asks  
                        (cost=0.43..15996.56 rows=143152 width=24) 
                        (actual time=0.027..0.027 rows=1 loops=57303)
                       Output: asks.value, asks.instrument, asks.ts
                       Index Cond: ((asks.instrument = bids.instrument) 
                          AND (asks.ts <= bids.ts))
 Planning Time: 1.231 ms
 Execution Time: 1703.821 msConclusionWhile PostgreSQL does not have anASOFkeyword, it does offer the flexibility and functionality to perform similar operations. When you're using Timescale, things only get better with the enhancements like Skip Scan.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/implementing-asof-joins-in-timescale/
2023-06-30T13:03:10.000Z,Nearest Neighbor Indexes: What Are ivfflat Indexes in pgvector and How Do They Work,"The rising popularity of ChatGPT, OpenAI, and applications of Large Language Models (LLMs) has brought the concept of approximate nearest neighbor search (ANN) to the forefront and sparked a renewed interest in vector databases due to the use of embeddings.Embeddingsare mathematical representations of phrases that capture the semantic meaning as a vector of numerical values.What makes this representation fascinating—and useful—is that phrases with similar meanings will have similar vector representations, meaning the distance between their respective vectors will be small. We recently discussed one application of these embeddings,Retrieval Augmented Generation—augmenting base LLMs with knowledge that it wasn’t trained on—but there are numerous other applications as well.One common application of embeddings is preciselysemantic similarity search. The basic concept behind this approach is that if I have a knowledge library consisting of various phrases and I receive a question from a user, I can locate the most relevant information in my library by finding the data that is most similar to the user's query.This is in contrast to lexical or full-text search, which only returns exact matches for the query. The remarkable aspect of this technique is that, since the embeddings represent the semantics of the phrase rather than its specific wording, I can find pertinent information even if it is expressed using completely different words!Semantic similarity search involves calculating an embedding for the user's question and then searching through my library to find the K most relevant items related to that question—these are the K items whose embeddings are closest to that of the question. However, when dealing with a large library, it becomes crucial to perform this search efficiently and swiftly. In the realm of vector databases, this problem is referred to as ""Finding the k nearest neighbors"" (KNN).This post discusses a method to enhance the speed of this search when utilizing PostgreSQL andpgvectorfor storing vector embeddings: theInverted File Flat (ivfflat)algorithm for approximate nearest neighbor search. We’ll cover why ivfflat is useful, how it works, and best practices for using it in pgvector for fast similarity search over embeddings vectors.Let’s go!Why Use the ivfflat Index in pgvector: The Curse of DimensionalitySearching for the k-nearest neighbors is not a novel problem for PostgreSQL.PostGIS, a PostgreSQL extension for handling location data, stores its data points as two-dimensional vectors (longitude and latitude). Locating nearby locations is a crucial query in that domain.PostGIS tackles this challenge by employing an index known as an R-Tree, which yields precise results for k-nearest neighbor queries. Similar techniques, such as KD-Trees and Ball Trees, are also employed for this type of search in other databases.However, there's a catch. These approaches cease to be effective when dealing with data larger than approximately 10 dimensions due to the ""curse of dimensionality."" Cue the ominous music! Essentially, as you add more dimensions, the available space increases exponentially, resulting in exponentially sparser data. This reduced density renders existing indexing techniques, like the aforementioned R-Tree, KD-Trees, and Ball Trees, which rely on partitioning the space, ineffective. (To learn more, I suggest these two videos:1,2).Given that embeddings often consist of more than a thousand dimensions—OpenAI’s are 1,536—new techniques had to be developed. There are no known exact algorithms for efficiently searching in such high-dimensional spaces. Nevertheless, there are excellentapproximatealgorithms that fall into the category of approximate nearest neighbor algorithms. Numerous such algorithms exist, but in this article, we will delve into the Inverted File Flat or ivfflat algorithm, which is provided by pgvector.How the ivfflat Index Works in pgvectorHow ivfflat divides the spaceTo gain an intuitive understanding of how ivfflat works, let's consider a set of vectors represented in a two-dimensional space as the following points:A set of vectors represented as points in two dimensionsIn the ivfflat algorithm, the first step involves applying k-means clustering to the vectors to find cluster centroids. In the case of the given vectors, let's assume we perform k-means clustering and identify four clusters with the following centroids.After k-means clustering, we identify four clusters indicated by the colored trianglesAfter computing the centroids, the next step is to assign each vector to its nearest centroid. This is accomplished by calculating the distance between the vector and each centroid and selecting the centroid with the smallest distance as the closest one. This process conceptually maps each point in space to the closest centroid based on proximity.By establishing this mapping, the space becomes divided into distinct regions surrounding each centroid (technically, this kind of division is called aVoronoi Diagram). Each region represents a cluster of vectors that exhibit similar characteristics or are close in semantic meaning.This division enables efficient organization and retrieval of approximate nearest neighbors during subsequent search operations, as vectors within the same region are likely to be more similar to each other than those in different regions.The process of assigning each vector to its closest centroid conceptually divides the space into distinct regions that surround each centroidBuilding the ivfflat index in pgvectorIvfflat proceeds to create aninverted indexthat maps each centroid to the set of vectors within the corresponding region. In pseudocode, the index can be represented as follows:inverted_index = {
  centroid_1: [vector_1, vector_2, ...],
  centroid_2: [vector_3, vector_4, ...],
  centroid_3: [vector_5, vector_6, ...],
  ...
}Here, each centroid serves as a key in the inverted index, and the corresponding value is a list of vectors that belong to the region associated with that centroid. This index structure allows for efficient retrieval of vectors in a region when performing similarity searches.Searching the ivfflat index in pgvectorLet's imagine we have a query for the nearest neighbors to a vector represented by a question mark, as shown below:We want to find nearest neighbors to the vector represented by the question markTo find the approximate nearest neighbors using ivfflat, the algorithm operates under the assumption that the nearest vectors will be located in the same region as the query vector. Based on this assumption, ivfflat employs the following steps:Calculate the distance between the query vector (red question mark) and each centroid in the index.Select the centroid with the smallest distance as the closest centroid to the query (the blue centroid in this example).Retrieve the vectors associated with the region corresponding to the closest centroid from the inverted index.Compute the distances between the query vector and each of the vectors in the retrieved set.Select the K vectors with the smallest distances as the approximate nearest neighbors to the query.The use of the index in ivfflat accelerates the search process by restricting the search to the region associated with the closest centroid. This results in a significant reduction in the number of vectors that need to be examined during the search. Specifically, if we have C clusters (centroids), on average, we can reduce the number of vectors to search by a factor of 1/C.Searching at the edgeThe assumption that the nearest vectors will be found in the same region as the query vector can introduce recall errors in ivfflat. Consider the following query:ivfflat can sometimes make errors when searching for nearest neighbors to a point at the edge of two regions of the vector spaceFrom visual inspection, it becomes apparent that one of the light-blue vectors is closer to the query vector than any of the dark-blue vectors, despite the query vector falling within the dark-blue region. This illustrates a potential error in assuming that the nearest vectors will always be found within the same region as the query vector.To mitigate this type of error, one approach is to search not only the region of the closest centroid but also the regions of the next closest R centroids. This approach expands the search scope and improves the chances of finding the true nearest neighbors.In pgvector, this functionality is implemented through the `probes` parameter, which specifies the number of centroids to consider during the search, as described below.Parameters for pgvector’s ivfflat ImplementationIn the implementation of ivfflat in pgvector, two key parameters are exposed: lists and probes.Lists parameter in pgvectorThelistsparameter determines the number of clusters created during index building (It’s called lists because each centroid has a list of vectors in its region). Increasing this parameter reduces the number of vectors in each list and results in smaller regions.It offers the following trade-offs to consider:Higherlistsvalue speeds up queries by reducing the search space during query time.However, it also decreases the region size, which can lead to more recall errors by excluding some points.Additionally, more distance comparisons are required to find the closest centroid during step one of the query process.Here are some recommendations for setting thelistsparameter:For datasets with less than one million rows, uselists =  rows / 1000.For datasets with more than one million rows, uselists = sqrt(rows).It is generally advisable to have at least 10 clusters.Probes parameter in pgvectorThe probes parameter is a query-time parameter that determines the number of regions to consider during a query. By default, only the region corresponding to the closest centroid is searched. By increasing the probes parameter, more regions can be searched to improve recall at the cost of query speed.The recommended value for the probes parameter isprobes = sqrt(lists).Using ivfflat in pgvectorCreating an indexWhen creating an index, it is advisable to have existing data in the table, as it will be utilized by k-means to derive the centroids of the clusters.The index in pgvector offers three different methods to calculate the distance between vectors: L2, inner product, and cosine. It is essential to select the same method for both the index creation and query operations. The following table illustrates the query operators and their corresponding index methods:Distance typeQuery operatorIndex methodL2 / Euclidean<->vector_l2_opsNegative Inner product<#>vector_ip_opsCosine<=>vector_cosine_opsNote: OpenAIrecommendscosine distance for its embeddings.To create an index in pgvector using ivfflat, you can use a statement using the following form:CREATE INDEX ON <table name> USING ivfflat (<column name> <index method>) WITH (lists = <lists parameter>);Replace<table name>with the name of your table and<column name>with the name of the column that contains the vector type.For example, if our table is namedembeddingsand our embedding vectors are in a column namedembedding, we can create an ivfflat index as follows:CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);Here’s a simple Python function that you can use to create an ivfflat index with the correct parameters forlistsandprobesas discussed above:def create_ivfflat_index(conn, table_name, column_name, query_operator=""<=>""): 
    index_method = ""invalid""
    if query_operator == ""<->"":
        index_method = ""vector_l2_ops""
    elif query_operator == ""<#>"":
        index_method = ""vector_ip_ops""
    elif query_operator == ""<=>"":
        index_method = ""vector_cosine_ops""
    else:
        raise ValueError(f""unrecognized operator {query_operator}"")

    with conn.cursor() as cur:
        cur.execute(f""SELECT COUNT(*) as cnt FROM {table_name};"")
        num_records = cur.fetchone()[0]

        num_lists = num_records / 1000
        if num_lists < 10:
            num_lists = 10
        if num_records > 1000000:
            num_lists = math.sqrt(num_records)

        cur.execute(f'CREATE INDEX ON {table_name} USING ivfflat ({column_name} {index_method}) WITH (lists = {num_lists});')
        conn.commit()QueryingAn index can be used whenever there is an ORDER BY of the formcolumn <query operator> <some pseudo-constant vector>along with a LIMIT k;Some examplesGet the closest two vectors to a constant vector:SELECT * FROM my_table ORDER BY embedding_column <=> '[1,2]' LIMIT 2;This is a common usage pattern in retrieval augmented generation using LLMs, where we find the embedding vectors that are closest in semantic meaning to the user’s query. In that case, the constant vector would be the embedding vector representing the user’s query.You can see an example of this in our guide tocreating, storing, and querying OpenAI embeddings with pgvector, where we use this Python function to find the three most similar documents to a given user query from our database:# Helper function: Get top 3 most similar documents from the database
def get_top3_similar_docs(query_embedding, conn):
    embedding_array = np.array(query_embedding)
    # Register pgvector extension
    register_vector(conn)
    cur = conn.cursor()
    # Get the top 3 most similar documents using the KNN <=> operator
    cur.execute(""SELECT content FROM embeddings ORDER BY embedding <=> %s LIMIT 3"", (embedding_array,))
    top3_docs = cur.fetchall()
    return top3_docsGet the closest vector to some row:SELECT * FROM my_table WHERE id != 1 ORDER BY embedding_column <=> (SELECT embedding_column FROM my_table WHERE id = 1) LIMIT 2;Tip:PostgreSQL's ability to use an index does not guarantee its usage! The cost-based planner evaluates query plans and may determine that a sequential scan or a different index is more efficient for a specific query. You can use the EXPLAIN command to see the chosen execution plan. To test the viability of using an index, you can modify planner costing parameters until you achieve the desired plan. For small datasets, settingenable_seqscan = 0can be especially advantageous for testing viability as it avoids sequential scans.To adjust the probes parameter, you can set theivfflat.probesvariable. For instance, to set it to '5', execute the following statement before running the query:SET ivfflat.probes = 5;Dealing with data changesAs your data evolves with inserts, updates, and deletes, the ivfflat index will be updated accordingly. New vectors will be added to the index, while no longer-used vectors will be removed.However, the clustering centroids will not be updated. Over time, this can result in a situation where the initial clustering, established during index creation, no longer accurately represents the data. This can be visualized as follows:As data gets inserted or deleted from the index, if the index is not rebuilt, the ivfflat index in pgvector can return incorrect approximate nearest neighbors due to clustering centroids no longer fitting the data wellTo address this issue, the only solution is to rebuild the index.Here are two important takeaways from this issue:Build the index once you have all representative data you want to reference in it, This is unlike most indexes which can be built on an empty table.It is advisable to periodically rebuild the index.When rebuilding the index, it is highly recommended to use the CONCURRENTLY option to avoid interfering with ongoing operations.Thus, to rebuild the index run the following in a cron job:REINDEX INDEX CONCURRENTLY <index name>;Summing It UpThe ivfflat algorithm in pgvector provides an efficient solution for approximate nearest neighbor search over high-dimensional data like embeddings. It works by clustering similar vectors into regions and building an inverted index to map each region to its vectors. This allows queries to focus on a subset of the data, enabling fast search. By tuning the lists and probes parameters, ivfflat can balance speed and accuracy for a dataset.Overall, ivfflat gives PostgreSQL the ability to perform fast semantic similarity search over complex data. With simple queries, applications can find the nearest neighbors to a query vector among millions of high-dimensional vectors. For natural language processing, information retrieval, and more, ivfflat is a compelling solution. By understanding how ivfflat divides the vector space into regions and builds its inverted index, you can optimize its performance for your needs and build powerful applications on top of it.✨Hands-on tutorials:Now that you know more about the ivfflat index in pgvector, get your hands dirty with using pgvector: Followour tutorialon creating, storing, and querying OpenAI embeddings using PostgreSQL as a vector database. Orlearn howto use pgvector as a vectorstore in LangChain.And if you’re looking for a production PostgreSQL database for your vector workloads,try Timescale. It’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/nearest-neighbor-indexes-what-are-ivfflat-indexes-in-pgvector-and-how-do-they-work/
2023-08-01T12:58:32.000Z,The 2023 State of PostgreSQL Survey Is Now Open!,"Almost half (45.55 %) of the2023 Stack Overflow Developer Surveyrespondents who answered the question about their favorite database (76,634 in total) chose PostgreSQL as the most popular one. This is a testament to the quality, reliability, and performance of PostgreSQL, as well as the vibrant and diverse community that supports it.As proud members of the PostgreSQL community, we want to continue giving back to this awesome group of data techies. We’re happy to announce that the 2023 State of PostgreSQL survey is officially live, and we are excited to hear once again from PostgreSQL users worldwide.Over the years, we have learned a lot about the community. In 2019, we noticed that while PostgreSQL is a popular choice among organizations,81 % of you use PostgreSQL for personal projects. In 2021,the community shared that the most frequently used extension is PostGIS. Last year,17 % of the respondents said they contributed to PostgreSQL at least once.This year, we want to know how these practices evolved, and we will explore the two magical letters of the hour—AI. We want to learn what AI tools the PostgreSQL community uses and if the AI workloads are already part of personal and work projects.The survey results and anonymized raw data will be published in a report that will be available for free to everyone. The report will provide valuable insights into the PostgreSQL ecosystem and help us understand how we can collectively make PostgreSQL better.For now, to whet the appetite for the 2023 report, read the highlights of last year’s findings. To download the full 2022 report, head over tohttps://www.timescale.com/state-of-postgres/2022/The survey is open until September 15, 2023.So what are you waiting for? Take the survey now and share your voice with the PostgreSQL community!Take the surveyThe State of PostgreSQL in 2022PostgreSQL's popularity is increasingThe number of PostgreSQL newbies using the database for less than a year has grown from 6.1 % in 2021 to 6.4 % in 2022.Reasons for choosing PostgreSQL over other databasesOpen-source, reliability, and extensions are the main reasons PostgreSQL users selected in 2022. Interestingly, users´ years of experience were directly related to their answers. “Reliability” was the number one reason to choose PostgreSQL among those who have been using the database for 11-15 years, while “open source” was primarily pointed out by users with up to five years of experience.PostgreSQL usage is growingSmall and medium businesses (0-50 employees) use PostgreSQL a lot more than they did one year ago. The result is on par with a broader trend: PostgreSQL’s usage is growing, with the majority of respondents—55 %—saying that they have increased their usage of the database.PostgreSQL users ♥ documentationThe majority of respondents (76.1 %) answered that technical documentation is their preferred way of learning about PostgreSQL, followed by long-form blog posts (51.5 %) and short-form blog posts (43.3 %). But the new generation of PostgreSQL enjoys learning slightly differently: users with less than five years of PostgreSQL experience gravitate toward video as their first option.PostgreSQL users increasingly use DBaaS providers to deploy PostgreSQLThe trend that we first saw in 2021 continues in 2022. Fewer PostgreSQL users reported self-managing the database compared to previous years. More respondents are using a managed PostgreSQL service to deploy the database.Many thanks to everyone who took the time to fill in the 2023 survey. If you have not done that yet, grab a cup or glass of your favorite beverage andshare your experience with PostgreSQL.Help us make the survey more representative. Share this far and wide! Post it in the company chat. Share it on social media.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/the-2023-state-of-postgresql-survey-is-now-open/
2023-07-12T13:00:15.000Z,How to Build LLM Applications With pgvector Vector Store in LangChain,"LangChain and pgvector: Up and RunningLangChainis one of the most popular frameworks for building applications and agents with Large Language Models (LLMs). This blog post is an introduction to building LLM applications with the LangChain framework in Python, using PostgreSQL and pgvector as a vector database for OpenAI embeddings data.We'll use the example of creating a chatbot to answer questions about the blog posts from the Timescale blog to illustrate the following concepts:How to prepare your documents for insertion into PostgreSQL and pgvector using LangChain document transformer TextSplitter.How to create embeddings from your data using the OpenAI embeddings modeland insert them into PostgreSQL and pgvector.How to use embeddings retrieved from a vector database to augment LLM generation.This is a great first step for more advanced LangChain projects in Python—for example, creating a chatbot for your company documentation or an application to answer questions from uploaded PDFs.Let's get started!💡Jupyter Notebook and Code:You can find all the code used in this tutorial in a Jupyter Notebook on GitHub in theTimescale Vector Cookbook repo. We recommend cloning the repo and following along by executing the code cells as you read through the tutorial.Setup and ConfigurationSign up for an OpenAI Developer Account and create an API Key. SeeOpenAI's developer platform.Install Python.Install and configure a Python virtual environment. We recommendpyenv.Install the requirements for this notebook using the following command:pip install -r requirements.txtOr, if you already have LangChain installed, runpip install --upgrade langchain.import os
# Run export OPENAI_API_KEY=sk-YOUR_OPENAI_API_KEY...
# Get openAI api key by reading local .env file
from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv())
OPENAI_API_KEY  = os.environ['OPENAI_API_KEY']Next, we need a way for LangChain to interact with PostgreSQL and pgvector. This is achieved by importing the PGVector class from thelangchain.vectorstorespackage as follows.from langchain.vectorstores.pgvector import PGVectorNext, we'll construct our connection string for LangChain to connect to our PostgreSQL database.Because LangChain usesSQLAlchemyto connect to SQL databases like PostgreSQL, we need to create our connection string programmatically, reading each of the components of the string (host, database name, password, port, etc.) from our environment variables.In this example, we'll use a PostgreSQL database with pgvector installed and hosted onTimescale. You can create your own cloud PostgreSQL database in minutesat this linkto follow along.If you're using a Timescale database, you can find all this information in the ""Cheat Sheet"" file you download when creating your new database service. Alternatively, you can also use a local PostgreSQL database if you prefer.# Build the PGVector Connection String from params
# Found in the credential cheat-sheet or ""Connection Info"" in the Timescale console
# In terminal, run: export VAR_NAME=value for each of the values below
host= os.environ['TIMESCALE_HOST']
port= os.environ['TIMESCALE_PORT']
user= os.environ['TIMESCALE_USER']
password= os.environ['TIMESCALE_PASSWORD']
dbname= os.environ['TIMESCALE_DBNAME']

# We use postgresql rather than postgres in the conn string since LangChain uses sqlalchemy under the hood
# You can remove the ?sslmode=require if you have a local PostgreSQL instance running without SSL
CONNECTION_STRING = f""postgresql+psycopg2://{user}:{password}@{host}:{port}/{dbname}?sslmode=require""Ensure you have the pgvector extension installed in your database. You can install it by running. Seesection 2.1 herefor how to install with psycopg2 and python if you prefer.CREATE EXTENSION IF NOT EXISTS vector;Part 1: Use LangChain to split a CSV file into smaller chunks while preserving associated metadataIn this section, we will parse our CSV file into smaller chunks for similarity search and retrieval, with help from LangChains TokenTextSplitter.First, let's take a look at the CSV file we'll be working with:import pandas as pd
import numpy as np
df = pd.read_csv('blog_posts_data.csv')
df.head()titlecontenturl0How to Build a Weather Station With Elixir, Ne...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-to-build-a-...1CloudQuery on Using PostgreSQL for Cloud Asset...This is an installment of our “Community Membe...https://www.timescale.com/blog/cloudquery-on-u...2How a Data Scientist Is Building a Time-Series...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-a-data-scie...3How Conserv Safeguards History: Building an En...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-conserv-saf...4How Messari Uses Data to Open the Cryptoeconom...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-messari-use...As shown above, this is a CSV file ofblog posts about Timescale use cases, in which the developers behind each project explain more about their data infra goals, how they used Timescale to achieve them and share success tips.Ordinarily, we would use the LangChainCSVLoaderto load the contents of a CSV file. But, in this case, we need to pre-process the content column of our CSV to be able to create embeddings for each blog post within the token limits of the OpenAI embeddings API.We also need a way to split the text of the content column of the CSV while retaining the associated metadata with that text (i.e., the blog title and URL).LangChain has several built-in document transformers that make it easy to split, combine, filter, and otherwise manipulate documents.We'll use LangChain'sToken Text Splitterto help us split up the content column of our CSV into chunks of a specified token amount. Alternatively, you can use theRecursive Character Text Splitterif you'd rather split text by the number of characters rather than tokens.We will split the text into chunks of around 512 tokens, with a 20 % or 103 token overlap.import tiktoken
from langchain.text_splitter import TokenTextSplitter
# Split text into chunks of 512 tokens, with 20% token overlap
text_splitter = TokenTextSplitter(chunk_size=512,chunk_overlap=103)Here’s how we’ll split up the chunks:# Helper func: calculate number of tokens
def num_tokens_from_string(string: str, encoding_name = ""cl100k_base"") -> int:
    if not string:
        return 0
    # Returns the number of tokens in a text string
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

#list for smaller chunked text and metadata
new_list = []

# Create a new list by splitting up text into token sizes of around 512 tokens
for i in range(len(df.index)):
    text = df['content'][i]
    token_len = num_tokens_from_string(text)
    if token_len <= 512:
        new_list.append([df['title'][i],
        df['content'][i], 
        df['url'][i]])
    else:
        #split text into chunks using text splitter
        split_text = text_splitter.split_text(text)
        for j in range(len(split_text)):
            new_list.append([df['title'][i],
            split_text[j],
            df['url'][i]])Let's take a look at how the content looks after being split:df_new = pd.DataFrame(new_list, columns=['title', 'content', 'url'])
df_new.head()Part 2: Insert OpenAI embeddings into PostgreSQL and pgvectorNow that we have our original CSV split up into smaller chunks and the associated metadata preserved, we will use the LangChainPandas DataFrame Loaderto load data from our new Pandas data frame and insert it into our PostgreSQL database with pgvector installed.Note that we must specify which column in the data frame contains the text we'll create embeddings for.#load documents from Pandas dataframe for insertion into database
from langchain.document_loaders import DataFrameLoader

# page_content_column is the column name in the dataframe to create embeddings for
loader = DataFrameLoader(df_new, page_content_column = 'content')
docs = loader.load()We'll use the OpenAI embeddings model for our documents, so let's import theOpenAIEmbeddingsmodule from thelangchain.embeddingspackage and create an instance.This instance can be used to generate embeddings for text data using the OpenAI API.from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()💡Learn more about embeddings:For more on OpenAI embeddings and how they're used in Nearest Neighbor Search, seethis explainer.Before we create embeddings for all the data in our data frame, let's briefly overview how creating an embedding works.Here's how we create an embedding for a string:# Create OpenAI embedding using LangChain's OpenAIEmbeddings class
query_string = ""PostgreSQL is my favorite database""
embed = embeddings.embed_query(query_string)
print(len(embed)) # Should be 1536, the dimensionality of OpenAI embeddings
print(embed[:5]) # Should be a list of floatsFor the main event, we'll connect to our PostgreSQL database and store the documents we loaded along with their embeddings.Thanks to LangChain, creating the embeddings and storing the data in our PostgreSQL database is a one-command operation!We pass in the following arguments:documents: The documents we loaded from the Pandas Data Frame.embedding: Our instance of the OpenAI embeddings class, the model we'll use to create the embeddings.collection_name: The name of the table we want our embeddings and metadata to live in.distance_strategy: The distance strategy we want to use to calculate the distance between vectors—in our case, we'll use Cosine distance.connection_string: The connection string to our PostgreSQL database, which we constructed in the setup section.# Create a PGVector instance to house the documents and embeddings
from langchain.vectorstores.pgvector import DistanceStrategy
db = PGVector.from_documents(
    documents= docs,
    embedding = embeddings,
    collection_name= ""blog_posts"",
    distance_strategy = DistanceStrategy.COSINE,
    connection_string=CONNECTION_STRING)Now that our data is in the database, let's perform asimilarity searchto fetch the documents most similar to a query:from langchain.schema import Document

# Query for which we want to find semantically similar documents
query = ""Tell me about how Edeva uses Timescale?""

#Fetch the k=3 most similar documents
docs =  db.similarity_search(query, k=3)The query on our database returns a list of LangChain documents, so let's learn how to interact with those:# Interact with a document returned from the similarity search on pgvector
doc = docs[0]

# Access the document's content
doc_content = doc.page_content
# Access the document's metadata object
doc_metadata = doc.metadata

print(""Content snippet:"" + doc_content[:500])
print(""Document title: "" + doc_metadata['title'])
print(""Document url: "" + doc_metadata['url'])Content snippet:map applications. If you are planning to store time-series data, Timescale is the way to go. It makes it easy to get started because it is “just” SQL, and at the same time, you get the important features needed to work with time-series data. I recommend you have a look, especially at continuous aggregations. Think about the whole lifecycle when you start. Will your use cases allow you to use features like compression, or do you need to think about how to store long-term data outside of TimescaleDBDocument title: How Edeva Uses Continuous Aggregations and IoT to Build Smarter CitiesDocument url:https://www.timescale.com/blog/how-edeva-uses-continuous-aggregations-and-iot-to-build-smarter-cities/Part 3: Question answering with Retrieval Augmented GenerationNext, let's tie everything we've learned together and build a simple example of using LangChain for questions answering using an LLM from OpenAI and the most relevant documents the question from our database.This technique is called Retrieval Augmented Generation (RAG) and works as follows:Create an embedding vector for the user question.Use pgvector to perform a vector similarity search and retrieve the k nearest neighbors to the question embedding from our database of embedding vectors representing the blog content. In our example, we’ll use k=3, finding the three most similar embedding vectors and associated content.Supply the content retrieved from the database as additional context to the model and ask it to perform a completion task to answer the user question.To more easily retrieve documents from our PostgreSQL vector database, we'll use a LangChainretriever.In LangChain, a retriever is an interface that returns documents given an unstructured query. A retriever's main purpose is only to return (or retrieve) documents.We will use avector store-backed retrieverwhich is a retriever that uses a vector store to retrieve documents. It is a lightweight wrapper around the Vector Store class to make it conform to the retriever interface. It uses the search methods implemented by a vector store, like similarity search, to query the texts in the vector store.# Create retriever from database
# We specify the number of results we want to retrieve (k=3)
retriever = db.as_retriever(
    search_kwargs={""k"": 3}
    )Next, we'll import the LLM we want to use to generate a response to our question. In this case, we'll useOpenAI's GPT-3.5 modelwith a 16k token context window so that we won't have any trouble fitting in retrieved documents as context in addition to the user question.from langchain.chat_models import ChatOpenAI
llm = ChatOpenAI(temperature = 0.0, model = 'gpt-3.5-turbo-16k')Then, we'll use one of the most useful chains in LangChain, theRetrieval Q+A chain, which is used for question answering over a vector database (vector store or index, as it’s also known).We'll combine it with astuff chain, which takes a list of documents, inserts them all into a prompt (stuffsthem in), and passes that prompt to an LLM.from langchain.chains import RetrievalQA
qa_stuff = RetrievalQA.from_chain_type(
    llm=llm, 
    chain_type=""stuff"", 
    retriever=retriever,
    verbose=True,
)And for the final ingredient, let's formulate a question we want the model to answer with the help of the documents in our database and pass it to our chain to process.query =  ""How does Edeva use continuous aggregates?""

response = qa_stuff.run(query)

from IPython.display import Markdown, display
display(Markdown(response))Edeva uses continuous aggregates in their smart city platform, EdevaLive. They collect large amounts of data from IoT devices, including traffic flow data from their dynamic speed bump called Actibump. Continuous aggregates allow them to roll up multiple resolutions of their sensor account data and people count data, making it available in a more efficient way. This helps them analyze and visualize the data faster, enabling them to provide valuable remote monitoring services and statistics to their customers. They also use continuous aggregates to roll up high-resolution data to lower resolutions, optimizing their data processing.Bonus: Cite Your Sources With LangChain and pgvector for RAGFor even more advanced functionality, you might want your answer to include the sources used to give users peace of mind. Here's how you can do that with the RetrievalQA chain using thereturn_source_documentsargument:# New chain to return context and sources
qa_stuff_with_sources = RetrievalQA.from_chain_type(
    llm=llm, 
    chain_type=""stuff"", 
    retriever=retriever,
    return_source_documents=True,
    verbose=True,
)

query =  ""How does Edeva use continuous aggregates?""

# To run the query, we use a different syntax since we're returning more than just the response text
responses = qa_stuff_with_sources({""query"": query})And finally, let's print out the result with the source document cited:source_documents = responses[""source_documents""]
source_content = [doc.page_content for doc in source_documents]
source_metadata = [doc.metadata for doc in source_documents]

# Construct a single string with the LLM output and the source titles and urls
def construct_result_with_sources():
    result = responses['result']
    result += ""\n\n""
    result += ""Sources used:""
    for i in range(len(source_content)):
    
    result += ""\n\n""
        result += source_metadata[i]['title']
        result += ""\n\n""
        result += source_metadata[i]['url']
    return result

display(Markdown(construct_result_with_sources()))Edeva uses continuous aggregates in their smart city platform, EdevaLive. They collect large amounts of data from IoT devices, including traffic flow data from their dynamic speed bump called Actibump. Continuous aggregates allow them to roll up multiple resolutions of their sensor account data and people count data, making it available in a more efficient way. This helps them analyze and visualize the data faster, enabling them to provide valuable remote monitoring services and statistics to their customers. They also use continuous aggregates to roll up high-resolution data to lower resolutions, optimizing their data processing.Sources used:How Edeva Uses Continuous Aggregations and IoT to Build Smarter Citieshttps://www.timescale.com/blog/how-edeva-uses-continuous-aggregations-and-iot-to-build-smarter-cities/How Density Manages Large Real Estate Portfolios Using TimescaleDBhttps://www.timescale.com/blog/density-measures-large-real-estate-portfolios-using-timescaledb/How Edeva Uses Continuous Aggregations and IoT to Build Smarter Citieshttps://www.timescale.com/blog/how-edeva-uses-continuous-aggregations-and-iot-to-build-smarter-cities/The “cite your sources” functionality is helpful because it can help explain unexpected responses from the model due to irrelevant but highly similar documents being retrieved from the database.Next Steps💡Try it yourself:You can find all the code used in this tutorial in a Jupyter Notebook on GitHub in theTimescale Vector Cookbook repo. Clone the repo and try running it yourself!Check outConversational Retrieval QA Chainfor adding memory and using what you learned in a chatbot conversation setting.Learn how pgvector finds approximate nearest neighbors in this blog post:What Are ivfflat Indexes in pgvector and How Do They Work.UseChainlitto build your own LLM chatbot in Python (Timescale tutorial coming soon!).And if you’re looking for a production PostgreSQL database for your vector workloads,try Timescale. It’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-build-llm-applications-with-pgvector-vector-store-in-langchain/
2023-06-21T18:22:10.000Z,"PostgreSQL as a Vector Database: Create, Store, and Query OpenAI Embeddings With pgvector","Looking for a “Hello world” tutorial for pgvector and OpenAI embeddings that gives you the basics of using PostgreSQL as a vector database? You’ve found it!Vector databases enable efficient storage and search of vector data and are essential to developing and maintaining AI applications using Large Language Models (LLMs).With a little help from thepgvector extension, you can leverage PostgreSQL, the flexible and robust SQL database, as a vector database to store and queryOpenAI embeddings. Used to measure the similarity of text strings, OpenAI embeddings are a type of data representation (in the shape of vectors, i.e., lists of numbers) for OpenAI’s models. Much more on OpenAI embeddings, pgvector and vector databases later in this post.We’ll use the example of creating a chatbot to answer questions about Timescale use cases, referencing content from theTimescale Developer Q&A blog posts, to illustrate the key concepts for creating, storing, and querying OpenAI embeddings with PostgreSQL and pgvector.We divided the post into three parts:Part 1: How to create embeddings from content using theOpenAI API.Part 2: How to use PostgreSQL as a vector database and store OpenAI embedding vectors using pgvector.Part 3: How to use embeddings retrieved from a vector database to augment LLM generation.One could think of this tutorial as a first step to building a chatbot that can reference a company knowledge base or developer docs.✨Jupyter Notebook and Code:You can find all the code used in this tutorial in a Jupyter Notebook, as well as sample content and embeddings on the Timescale GitHub:timescale/vector-cookbook. We recommend cloning the repo and following along by executing the code cells as you read through the tutorial.Why Create and Store OpenAI Embeddings for Your Documents?We will explain how to useRetrieval Augmented Generation (RAG)to create a chatbot that combines your data with the power of ChatGPT using OpenAI and pgvector. RAG addresses the problem that a foundational model (e.g., GPT-3 or GPT-4) may be missing some information needed to give a good answer because that information was not in the dataset used to train the model (for example, the information is stored in private documents or only became available recently).RAG’s solution is dead simple: provide additional context to the foundational model in the prompt. For example, if someone asks a baking chatbot, “What is a cronut?” and the foundational model has never heard of cronuts, you can transform the prompt into context: “A cronut resembles a doughnut and is made from croissant-like dough filled with flavored cream and fried in grapeseed oil. What is a cronut?”The foundational model can then use its knowledge of donuts and croissants to wax eloquently about cronuts. This technique is insanely powerful—it allows you to “teach” foundational models about things only you know about and use that to create a ChatGPT++ experience for your users!But what context do you provide to the model? If you have a library of information, how do you know what’s relevant to a given question? Cue in embeddings. As mentioned above,OpenAI embeddingsare a mathematical representation of the semantic meaning of a piece of text that allows forsimilarity search.This means that if you get a user question and calculate its embedding, you can use similarity search against data embeddings in your library to find the most relevant information. But that requires having an embedding representation of your library.This post is a guide to creating, storing, and querying OpenAI vector embeddings usingpgvector, the extension that turns PostgreSQL into a vector database.What is pgvector?Pgvectoris an open-source extension for PostgreSQL that enables storing and searching over machine learning-generated embeddings. It provides different capabilities that let users identify both exact and approximate nearest neighbors. It is designed to work seamlessly with other PostgreSQL features, including indexing and querying.Let's get started!Before You Begin: Pre-Requisites and ConfigurationInstall Python.Install and configure a Python virtual environment. We recommendPyenv.Install the requirements for this notebook using the following command:pip install -r requirements.txtImport all the packages we will be using:import openai
import os
import pandas as pd
import numpy as np
import json
import tiktoken
import psycopg2
import ast
import pgvector
import math
from psycopg2.extras import execute_values
from pgvector.psycopg2 import register_vectorYou’ll need tosign up for an OpenAI Developer Accountand create an OpenAI API Key – we recommend getting a paid account to avoid rate limiting and settting a spending cap so that you avoid any surprises with bills.Once you have an OpenAI API key, it’s abest practiceto store it as an environment variable and then have your Python program read it.#First, run export OPENAI_API_KEY=sk-YOUR_OPENAI_API_KEY...


# Get openAI api key by reading local .env file
from dotenv import load_dotenv, find_dotenv
_ = load_dotenv(find_dotenv()) 
openai.api_key  = os.environ['OPENAI_API_KEY']Part 1: Create Embeddings for Your Data Using the OpenAI APIEmbeddingsmeasure how related text strings are. First, we'll create embeddings using the OpenAI API on some text we want the LLM to answer questions on.In this example, we'll use content from the Timescale blog, specifically from theDeveloper Q&A section, which features posts by Timescale users talking about their real-world use cases.You can replace this blog data with any text you want to embed, such as your own company blog, developer documentation, internal knowledge base, or any other information you’d like to have a “ChatGPT-like” experience over.# Load your CSV file into a pandas DataFrame
df = pd.read_csv('blog_posts_data.csv')
df.head()The output looks like this:TitleContentURL0How to Build a Weather Station With Elixir, Ne...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-to-build-a-...1CloudQuery on Using PostgreSQL for Cloud Asset...This is an installment of our “Community Membe...https://www.timescale.com/blog/cloudquery-on-u...2How a Data Scientist Is Building a Time-Series...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-a-data-scie...3How Conserv Safeguards History: Building an En...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-conserv-saf...4How Messari Uses Data to Open the Cryptoeconom...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-messari-use...1.1 Calculate the cost of embedding dataIt's usually a good idea to calculate how much creating embeddings for your selected content will cost. We provide a number of helper functions to calculate a cost estimate before creating the embeddings to help us avoid surprises.For OpenAI, you are charged on a per-token basis for embeddings created. The total cost will be less than $0.01 for the blog posts we want to embed, thanks to OpenAI’s recent announcement of a75 % cost reductionin their most popular embedding model,text-embedding-ada-002.What is a token?Tokens are common sequences of characters found in text. Roughly speaking, a token is three-quarters (¾) of a word. Large language models, like GPT-3 and GPT-4 made by OpenAI, are trained to understand the statistical relationships between tokens and predict the next token in a sequence. Learn more about tokens withOpenAI’s Tokenizer tool.# Helper functions to help us create the embeddings

# Helper func: calculate number of tokens
def num_tokens_from_string(string: str, encoding_name = ""cl100k_base"") -> int:
    if not string:
        return 0
    # Returns the number of tokens in a text string
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

# Helper function: calculate length of essay
def get_essay_length(essay):
    word_list = essay.split()
    num_words = len(word_list)
    return num_words

# Helper function: calculate cost of embedding num_tokens
# Assumes we're using the text-embedding-ada-002 model
# See https://openai.com/pricing
def get_embedding_cost(num_tokens):
    return num_tokens/1000*0.0001

# Helper function: calculate total cost of embedding all content in the dataframe
def get_total_embeddings_cost():
    total_tokens = 0
    for i in range(len(df.index)):
        text = df['content'][i]
        token_len = num_tokens_from_string(text)
        total_tokens = total_tokens + token_len
    total_cost = get_embedding_cost(total_tokens)
    return total_cost# quick check on total token amount for price estimation
total_cost = get_total_embeddings_cost()
print(""estimated price to embed this content = $"" + str(total_cost))1.2 Create smaller chunks of contentThe OpenAI API has alimitto the maximum number of tokens it can create an embedding for in a single request: 8,191 to be specific.To get around this limit, we'll break up our text into smaller chunks. Generally, it's a best practice to “chunk” the documents you want to create embeddings into groups of a fixed token size.The precise number of tokens to include in a chunk depends on your use case and your model’s context window—the number of input tokens it can handle in a prompt.For our purposes, we'll aim for chunks of around 512 tokens each. Chunking text up is a complex topic worthy of its own blog post. We’ll illustrate a simple method we found to work well below. If you want to read about other approaches, we recommendthis blog postandthis sectionof the LangChain docs.Note:If you prefer to skip this step, you can use the provided file:blog_data_and_embeddings.csv, which contains the data and embeddings that you'll generate in this step.The code below creates a new list of our blog content while retaining the metadata associated with the text, such as the blog title and URL that the text is associated with.# Create new list with small content chunks to not hit max token limits
# Note: the maximum number of tokens for a single request is 8191
# https://openai.com/docs/api-reference/requests

# list for chunked content and embeddings
new_list = []
# Split up the text into token sizes of around 512 tokens
for i in range(len(df.index)):
    text = df['content'][i]
    token_len = num_tokens_from_string(text)
    if token_len <= 512:
        new_list.append([df['title'][i], df['content'][i], df['url'][i], token_len])
    else:
        # add content to the new list in chunks
        start = 0
        ideal_token_size = 512
        # 1 token ~ 3/4 of a word
        ideal_size = int(ideal_token_size // (4/3))
        end = ideal_size
        #split text by spaces into words
        words = text.split()

        #remove empty spaces
        words = [x for x in words if x != ' ']

        total_words = len(words)
        
        #calculate iterations
        chunks = total_words // ideal_size
        if total_words % ideal_size != 0:
            chunks += 1
        
        new_content = []
        for j in range(chunks):
            if end > total_words:
                end = total_words
            new_content = words[start:end]
            new_content_string = ' '.join(new_content)
            new_content_token_len = num_tokens_from_string(new_content_string)
            if new_content_token_len > 0:
                new_list.append([df['title'][i], new_content_string, df['url'][i], new_content_token_len])
            start += ideal_size
            end += ideal_sizeNow that our text is chunked better, we can create embeddings for each chunk of text using the OpenAI API.We’ll use this helper function to create embeddings for a piece of text:# Helper function: get embeddings for a text
def get_embeddings(text):
   response = openai.Embedding.create(
       model=""text-embedding-ada-002"",
       input = text.replace(""\n"","" "")
   )
   embedding = response['data'][0]['embedding']
   return embeddingAnd then create embeddings for each chunk of content:# Create embeddings for each piece of content
for i in range(len(new_list)):
   text = new_list[i][1]
   embedding = get_embeddings(text)
   new_list[i].append(embedding)

# Create a new dataframe from the list
df_new = pd.DataFrame(new_list, columns=['title', 'content', 'url', 'tokens', 'embeddings'])
df_new.head()The new data frame should look like this:TitleContentURLTokensEmbeddings0How to Build a Weather Station With Elixir, Ne...This is an installment of our “Community Membe...https://www.timescale.com/blog/how-to-build-a-...501[0.021440856158733368, 0.02200360782444477, -0...1How to Build a Weather Station With Elixir, Ne...capture weather and environmental data. In all...https://www.timescale.com/blog/how-to-build-a-...512[0.016165969893336296, 0.011341351084411144, 0...2How to Build a Weather Station With Elixir, Ne...command in their database migration:SELECT cre...https://www.timescale.com/blog/how-to-build-a-...374[0.022517921403050423, -0.0019158280920237303,...3CloudQuery on Using PostgreSQL for Cloud Asset...This is an installment of our “Community Membe...https://www.timescale.com/blog/cloudquery-on-u...519[0.009028822183609009, -0.005185891408473253, ...4CloudQuery on Using PostgreSQL for Cloud Asset...Architecture with CloudQuery SDK- Writing plug...https://www.timescale.com/blog/cloudquery-on-u...511[0.02050386555492878, 0.010169642977416515, 0....As an optional but recommended step, you can save the original blog content along with associated embeddings in a CSV file for reference later on so that you don't have to recreate embeddings if you want to reference it in another project.# Save the dataframe with embeddings as a CSV file
df_new.to_csv('blog_data_and_embeddings.csv', index=False)Part 2: Store OpenAI Embeddings in a Vector Database Using pgvectorNow that we have created embedding vectors for our blog content, the next step is to store the embedding vectors in a vector database to help us perform a fast search over many vectors.What is a vector database?A vector database is a database that can handle vector data.Vector databases are useful for:Semantic search:Vector databases facilitate semantic search, which considers the context or meaning of search terms rather than just exact matches. They are useful for recommendation systems, content discovery, and question-answering systems.Efficient similarity search:Vector databases are designed for efficient high-dimensional nearest neighbor search, a task where traditional relational databases struggle.Machine learning:Vector databases store and search embeddings created by machine-learning models. This feature aids in finding items semantically similar to a given item.Multimedia data handling:Vector databases also excel in working with multimedia data (images, audio, video) by converting them into high-dimensional vectors for efficient similarity search.NLP and data combination:In Natural Language Processing (NLP), vector databases store high-dimensional vectors representing words, sentences, or documents. They also allow a combination of traditional SQL queries with similarity searches, accommodating both structured and unstructured data.We’ll use PostgreSQL with thepgvector extensioninstalled as our vector database. Pgvector extends PostgreSQL to handle vector data types and vector similarity search, likenearest neighbor search, which we’ll use to find thekmost related embeddings in our database for a given user prompt.Why use pgvector as a vector database?Here are five reasons why PostgreSQL is a good choice for storing and handling vector data:Integrated solution:By using PostgreSQL as a vector database, you keep your data in one place. This can simplify your architecture by reducing the need for multiple databases or additional services.Enterprise-level robustness and operations:With a 30-year pedigree, PostgreSQL provides world-class data integrity, operations, and robustness. This includes backups, streaming replication, role-based and row-level security, and ACID compliance.Full-featured SQL:PostgreSQL supports a rich set of SQL features, including joins, subqueries, window functions, and more. This allows for powerful and complex queries that can include both traditional relational data and vector data. It also integrates with a plethora of existing data science and data analysis tools.Scalability and performance:PostgreSQL is known for its robustness and ability to handle large datasets. Using it as a vector database allows you to leverage these characteristics for vector data as well.Open source:PostgreSQL is open source, which means it's free to download and use, and you can modify it to suit your needs. It also means that it benefits from the collective input of developers all over the world, which often results in high-quality, secure, and up-to-date software. PostgreSQL has a large and active community, so help is readily available. There are many resources, such as documentation, tutorials, forums, and more, to help you troubleshoot and optimize your PostgreSQL database.2.1 Create a PostgreSQL database and install pgvectorFirst, we’ll create a PostgreSQL database. You cancreate a cloud PostgreSQL databasein minutes for free onTimescaleor use a local PostgreSQL database for this step.Once you’ve created your PostgreSQL database, export your connection string as an environment variable, and just like the OpenAI API key, we’ll read it into our Python program from the environment file:# Timescale database connection string
# Found under ""Service URL"" of the credential cheat-sheet or ""Connection Info"" in the Timescale console
# In terminal, run: export TIMESCALE_CONNECTION_STRING=postgres://<fill in here>

connection_string  = os.environ['TIMESCALE_CONNECTION_STRING']We then connect to our database using the popularpsycopg2python library  and install the pgvector extension as follows:# Connect to PostgreSQL database in Timescale using connection string
conn = psycopg2.connect(connection_string)
cur = conn.cursor()

#install pgvector
cur.execute(""CREATE EXTENSION IF NOT EXISTS vector"");
conn.commit()2.2 Connect to and configure your vector databaseOnce we’ve installed pgvector, we use theregister_vector()command to register the vector type with our connection:# Register the vector type with psycopg2
register_vector(conn)Once we’ve connected to the database, let’s create a table that we’ll use to store embeddings along with metadata. Our table will look as follows:idtitleurlcontenttokensembeddingIdrepresents the unique ID of each vector embedding in the table.titleis the blog title from which the content associated with the embedding is taken.urlis the blog URL from which the content associated with the embedding is taken.contentis the actual blog content associated with the embedding.tokensis the number of tokens the embedding represents.embeddingis the vector representation of the content.One advantage of using PostgreSQL as a vector database is that you can easily store metadata and embedding vectors in the same database, which is helpful for supplying the user-relevant information related to the response they receive, like links to read more or specific parts of a blog post that are relevant to them.# Create table to store embeddings and metadata
table_create_command = """"""
CREATE TABLE embeddings (
            id bigserial primary key, 
            title text,
            url text,
            content text,
            tokens integer,
            embedding vector(1536)
            );
            """"""

cur.execute(table_create_command)
cur.close()
conn.commit()2.3 Ingest and store vector data into PostgreSQL using pgvectorNow that we’ve created the database and created the table to house the embeddings and metadata, the final step is to insert the embedding vectors into the database.For this step, it’s a best practice to batch insert the embeddings rather than insert them one by one.#Batch insert embeddings and metadata from dataframe into PostgreSQL database
register_vector(conn)
cur = conn.cursor()
# Prepare the list of tuples to insert
data_list = [(row['title'], row['url'], row['content'], int(row['tokens']), np.array(row['embeddings'])) for index, row in df_new.iterrows()]
# Use execute_values to perform batch insertion
execute_values(cur, ""INSERT INTO embeddings (title, url, content, tokens, embedding) VALUES %s"", data_list)
# Commit after we insert all embeddings
conn.commit()Let’s sanity check by running some simple queries against our newly inserted data:cur.execute(""SELECT COUNT(*) as cnt FROM embeddings;"")
num_records = cur.fetchone()[0]
print(""Number of vector records in table: "", num_records,""\n"")
# Correct output should be 129# print the first record in the table, for sanity-checking
cur.execute(""SELECT * FROM embeddings LIMIT 1;"")
records = cur.fetchall()
print(""First record in table: "", records)2.4 Index your data for faster retrievalIn this example, we only have 129 embedding vectors, so searching through all of them is blazingly fast. But for larger datasets, you need to create indexes to speed up searching for similar embeddings, so we include the code to build the index for illustrative purposes.Pgvector supports the ivfflat index type to provide for speed up of approximate nearest neighbor (ANN) searches (similarity search indexes for high-dimensionality data is very often approximate).You always want to build this indexafteryou have inserted the data, as the index needs to discover clusters in your data to be effective, and it does this only when first building the index.The index has a tunable parameter of the number of lists to use, and the code below shows the best practice for tuning this parameter. You also need to specify the distance measure used for indexing and ensure it matches the measure you use in your queries. In our case, we use the Cosine distance for querying below, and so we create our index withvector_cosine_ops.# Create an index on the data for faster retrieval

#calculate the index parameters according to best practices
num_lists = num_records / 1000
if num_lists < 10:
   num_lists = 10
if num_records > 1000000:
   num_lists = math.sqrt(num_records)

#use the cosine distance measure, which is what we'll later use for querying
cur.execute(f'CREATE INDEX ON embeddings USING ivfflat (embedding vector_cosine_ops) WITH (lists = {num_lists});')
conn.commit()Part 3: Nearest Neighbor Search Using pgvectorGiven a user question, we’ll perform the following steps to use information stored in the vector database to answer their question using Retrieval Augmented Generation:Create an embedding vector for the user question.Use pgvector to perform a vector similarity search and retrieve theknearest neighbors to the question embedding from our embedding vectors representing the blog content. In our example, we’ll use k=3, finding the three most similar embedding vectors and associated content.Supply the content retrieved from the database as additional context to the model and ask it to perform a completion task to answer the user question.3.1 Define a question you want to answerFirst, we’ll define a sample question that a user might want to answer about the blog posts stored in the database.# Question about Timescale we want the model to answer
input = ""How is Timescale used in IoT?""Since Timescale ispopular for IoT sensor data, a user might want to learn specifics about how they can leverage it for that use case.3.2 Find the most relevant content in the databaseHere’s the function we use to find the three nearest neighbors to the user question. Note it uses pgvector’s<=>operator, which finds theCosine distance(also known as Cosine similarity) between two embedding vectors.# Helper function: Get top 3 most similar documents from the database
def get_top3_similar_docs(query_embedding, conn):
    embedding_array = np.array(query_embedding)
    # Register pgvector extension
    register_vector(conn)
    cur = conn.cursor()
    # Get the top 3 most similar documents using the KNN <=> operator
    cur.execute(""SELECT content FROM embeddings ORDER BY embedding <=> %s LIMIT 3"", (embedding_array,))
    top3_docs = cur.fetchall()
    return top3_docs3.3 Define helper functions to query OpenAIWe supply helper functions to create an embedding for the user question and to get a completion response from an OpenAI model. We use GPT-3.5, but you can use GPT-4 or any other model from OpenAI.We also specify a number of parameters, such as limits of the maximum number of tokens in the model response and model temperature, which controls the randomness of the model, which you can modify to your liking:# Helper function: get text completion from OpenAI API
# Note we're using the latest gpt-3.5-turbo-0613 model
def get_completion_from_messages(messages, model=""gpt-3.5-turbo-0613"", temperature=0, max_tokens=1000):
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=temperature, 
        max_tokens=max_tokens, 
    )
    return response.choices[0].message[""content""]

# Helper function: get embeddings for a text
def get_embeddings(text):
    response = openai.Embedding.create(
        model=""text-embedding-ada-002"",
        input = text.replace(""\n"","" "")
    )
    embedding = response['data'][0]['embedding']
    return embedding3.3 Putting it all togetherWe’ll define a function to process the user input by retrieving the most similar documents from our database and passing the user input, along with the relevant retrieved context to the OpenAI model to provide a completion response to.Note that we modify the system prompt as well in order to influence the tone of the model’s response.We pass to the model the content associated with the three most similar embeddings to the user input using the assistant role. You can also append the additional context to the user message.# Function to process input with retrieval of most similar documents from the database
def process_input_with_retrieval(user_input):
    delimiter = ""```""

    #Step 1: Get documents related to the user input from database
    related_docs = get_top3_similar_docs(get_embeddings(user_input), conn)

    # Step 2: Get completion from OpenAI API
    # Set system message to help set appropriate tone and context for model
    system_message = f""""""
    You are a friendly chatbot. \
    You can answer questions about timescaledb, its features and its use cases. \
    You respond in a concise, technically credible tone. \
    """"""

    # Prepare messages to pass to model
    # We use a delimiter to help the model understand the where the user_input starts and ends
    messages = [
        {""role"": ""system"", ""content"": system_message},
        {""role"": ""user"", ""content"": f""{delimiter}{user_input}{delimiter}""},
        {""role"": ""assistant"", ""content"": f""Relevant Timescale case studies information: \n {related_docs[0][0]} \n {related_docs[1][0]} {related_docs[2][0]}""}   
    ]

    final_response = get_completion_from_messages(messages)
    return final_responseLet’s see an example of the model’s output to our original input question:input = ""How is Timescale used in IoT?""
response = process_input_with_retrieval(input)
print(input)
print(response)Here’s the model’s response:User input: How is Timescale used in IoT?Model response:TimescaleDB is commonly used in IoT (Internet of Things) applications for storing and analyzing time-series data generated by IoT devices. IoT devices generate a large volume of data over time, such as sensor readings, device status updates, and alarm information. TimescaleDB provides a scalable and efficient solution for storing and querying this time-series data.With TimescaleDB, IoT applications can benefit from features such as automatic data partitioning, compression, and retention policies. These features allow for efficient storage and retrieval of time-series data, even as the data volume grows. TimescaleDB also supports SQL queries, making it easy to perform complex analytics and aggregations on the time-series data.In IoT applications, TimescaleDB can be used to monitor and analyze various aspects, such as environmental conditions (temperature, humidity), energy consumption, equipment performance, and predictive maintenance. It enables real-time monitoring, anomaly detection, and trend analysis, helping businesses make data-driven decisions and optimize their operations.Overall, TimescaleDB provides a reliable and scalable database solution for handling the high-volume time-series data generated by IoT devices, enabling businesses to unlock valuable insights and improve their IoT applications.We can also ask the model questions about specific documents in the database, in this case about specific Timescale users who have spoken about their experience in a blog post:input_2 = ""Tell me about Edeva and Hopara. How do they use Timescale?""
response_2 = process_input_with_retrieval(input_2)
print(input_2)
print(response_2)Here’s the model output for the input question above; notice how it uses specific details from the blog posts.User input: Tell me about Edeva and Hopara. How do they use Timescale?Model response:Edeva and Hopara are two companies that utilize TimescaleDB for their data storage and visualization needs.Edeva uses TimescaleDB as the main database in their smart city system. Their clients can control their IoT devices and view captured data, gaining insights from trends and historical data. They leverage TimescaleDB's continuous aggregations feature to speed up queries and improve dashboard performance.Hopara, on the other hand, provides a sophisticated visualization system that allows users to derive insights from various types of data. They use TimescaleDB to store real-time vibration data from sensor-tagged machines. Hopara's visualization system, powered by TimescaleDB, enables users to drill down into the data and identify vibration issues.Both Edeva and Hopara benefit from TimescaleDB's time-series functionality and its ability to handle large amounts of data efficiently.ConclusionRetrieval Augmented Generation (RAG) is a powerful method of building applications with LLMs that enable you to teach foundation models about things it was not originally trained on—like private documents or recently published information.We covered the basics of creating a chatbot to answer questions about a blog. We used the content from the Timescale Developer Q&A blog posts as an example to show how to create, store, and perform similarity search on OpenAI embeddings.We used PostgreSQL and pgvectoras our vector database to store and query the embeddings.✨Jupyter Notebook and Code:You can find all the code used in this tutorial in a Jupyter Notebook, as well as sample content and embeddings on the Timescale GitHub:timescale/vector-cookbook.And if you’re looking for a production PostgreSQL database for your vector workloads,try Timescale. It’s free for 30 days, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/postgresql-as-a-vector-database-create-store-and-query-openai-embeddings-with-pgvector/
2023-02-07T14:00:00.000Z,Timescale Timeout: The History and Basics of PostgreSQL,"Welcome to the second installment of Timescale Timeout, a series of dev-to-dev talks about data-intensive applications, from individual projects to industry technologies, from best moments to lessons learned by cloud-native application developers.In this two-part episode, I spoke to my colleague and fellow Timescale developer advocate, Kirk Roybal. Together, we divedinto the history of PostgreSQL, how it has evolved over the years, its use cases and competitive advantages over other databases, and so much more.Kirk is a great storyteller and, in his own words, a genuine “PostgreSQL guy.” So, if you’re interested in learning more about how PostgreSQL works under the hood, its origins, and cool tips for users of all levels,including useful extensions, check this out. It will be worth your time.In the second episode, we carried on our conversation about the history of PostgreSQL by discussing the birth of extensions, tips on avoiding common beginner mistakes, and book recommendations.Become a better data-intensive application developer and get more stories like this delivered directly to your inboxby subscribing to our newsletter.(This post was originally published in late January 2023 with only the first video and updated in early February 2023 to include both episodes.)Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescale-timeout-the-history-and-basics-of-postgresql-part-i/
2023-11-03T11:12:50.000Z,Boosting Postgres Performance With Prepared Statements and PgBouncer's Transaction Mode,"Adoptingprepared statementsin your application is an easy performance gain. Using prepared statements lets your application skip query parsing and analyzing, eliminating a substantial amount of overhead. Pairing this with a connection pooler and transaction mode can dramatically boost your Postgres database performance. Needless to say, we were excited to learn that support for prepared statements in transaction mode was introduced in the1.21 release of PgBouncer!In this post we’ll cover, at a high level, why you should enable prepared statements in your application. If you’re looking for the gory details of the implication,our peers over at Crunchy wrote a great post on the 1.21 release here!Connection poolers on our mature cloud platform, Timescale, use PgBouncerunder the hood, supporting prepared statements and transaction pooling.Start a free trial today to try it out—no credit card required!What is a prepared statement, and why does it boost Postgres’ performance?Aprepared statementis a query that can accept parameters that youPREPAREon the server side, which does the query parsing, analysis, and any rewriting. You can then call this query byEXECUTEing it with the corresponding arguments.I like to think of prepared statements as similar to functions. You create a template of what will happen in the database and can call that template with parameters to have it happen inside the database. To over-extend the analogy, this “template” (prepared statements) lets you pre-compile the query, potentially greatly improving the overall execution time.In an application setting, many of your queries are probably already templated rather than executing arbitrary queries. These are ideal candidates to use as prepared statements since they repeat the same underlying query but with different values.How to make it work with my app?One limitation of the implementation of prepared statement support in PgBouncer is that “PgBouncer tracks protocol-level named prepared statements.” Basically, rather than writing raw SQL, we should use the libpq implementation of prepared statements instead.Fortunately for you, whatever language your application is likely using already relies on this as part of the object-relational mapping (ORM) implementation! We just need to use the corresponding API rather than writingPREPARE … EXECUTE …in raw SQL ourselves.ActiveRecord, the default ORM of Rails, makes this even simpler for us by using prepared statements by default. In yourconfig/database.ymlfile, ensure you have not altered your production environment to turn offprepared_statements. They will allow up to 1,000 prepared statements by default.Since Timescale allows for 100, we recommend reducing the config to equal ourmax_prepared_statements(see the next section for more detail). Thus, your config might look something like this:production:
  adapter: postgresql
  statement_limit: 100Note thatprepared_statements: falseis absent, as we want them on (which they are by default).For an example of what is happening under the hood or to use as a template for other ORMs that may not handle this automatically, in the Ruby pg gem, we have theprepare()function. Creating a prepared statement would look something like:conn = PG.connect(:dbname => 'tsdb')
conn.prepare('statement1', 'insert into metrics (created, type_id, value) values ($1::timestamptz, $2::int, $3::float)')This uses the same table structure as ourEnergy tutorial. Note that theofficial gem documentationrecommends casting the values to the desired types to avoid type conflicts. These are SQL types, not Ruby types, though, since they are a part of the query.To execute the query, you’d use something like:conn.exec_prepared('statement1', [ 2023-05-31 23:59:59.043264+00,13, 1.78 ])As a quirk of the PgBouncer implementation, we do not need toDEALLOCATEprepared statements. PgBouncer handles this automatically for us.All you need to do is, for each connection, try to prepare the statement you want to use once and then callexec_prepared()as often as the connection stays open.What’s happening behind the scenes?Behind the scenes, PgBouncer intercepts the creation of the prepared statement and creates a cache of prepared statements for the connection pool. It looks at what the client called it (statement1in our example) and sees if it has a prepared statement already on that connection, which looks something likePGBOUNCER_1234. If it does, then it will just reuse the already created statement. If not, it will automatically handle this and create it on behalf of the client in the database connection.This implementation lets you effectively cache prepared statements across connections —which is insanely cool—giving you the full benefit of prepared statements even when using a transaction pool. For example, if your pool size is 20, and you have 100 client connections, each running the same query, this means that, at most, the query will be planned 20 times instead of the standard 100 in previous transaction mode usage. That’s pretty awesome!One thing to be mindful of is themax_prepared_statementsset in your connection pooler. On Timescale, the default is 100. For ideal performance, it’s recommended to keep the number of different prepared statements your application uses to be less than this value.This lets you get maximum efficiency of PgBouncer’s cache. If you go more than this, it is not a big deal, but it may result in slightly more query plans than otherwise, as prepared statements will get deallocated from the database connection. For example, on Timescale, with your 101st prepared statement, the first prepared statement will be replaced by it.📚Need some advice on how to configure PgBouncer correctly? Here you go!Final StatementsPrepared statements rock! Using them can be a pretty easy performance win for your application while also boosting your Postgres database performance. The only thing to be careful about is to make sure that your application is using the libpq version of creating a prepared statement. Basically, rather than writing the raw SQL yourself, make sure you use your ORM’s API to create a prepared statement! Also, in PgBouncer, you don’t have to worry about deallocating a prepared statement ever.If you’d like to try this out yourself,Timescale offers a free trial—no credit card required!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/boosting-postgres-performance-with-prepared-statements-and-pgbouncers-transaction-mode/
2023-08-31T14:16:35.000Z,Making PostgreSQL Backups 100x Faster via EBS Snapshots and pgBackRest,"If you have experience running PostgreSQL in a production environment, you know that maintaining database backups is a daunting task. In the event of a catastrophic failure, data corruption, or other form of data loss, the ability to quickly restore from these backups will be vital for minimizing downtime. If you’re managing a database, maintaining your backups and getting your recovery strategy in order is probably the first check on your checklist.Perhaps this has already given you one headache or two becausecreating and restoring backups for large PostgreSQL databases can be a very slow process.The most widely used external tool for backup operations in PostgreSQL ispgBackRest, which is very powerful and reliable. But pgBackRest can also be very time-consuming, especially for databases well over 1 TB.The problem is exacerbated when restoring backups from production databases that continue to ingest data, thus creating more WAL (write-ahead log) that must be applied. In this case, a full backup and restore can take hours or even days, which can be a nightmare in production databases.When operating our platform (Timescale, a cloud database platform built on PostgreSQL), we struggled with this very thing. At Timescale, we pride ourselves in making PostgreSQL faster and more scalable for large volumes of time-series data—therefore, our customers’ databases are often large (many TBs). At first, we were completely basing our backup and restore operations in pgBackRest, and we were experiencing some pain:Creating full backups was very slow. This was a problem, for example, when our customers were trying to upgrade their PostgreSQL major version within our platform, as we took a fresh, full backup after upgrade in case there was a failure shortly after. Upgrades are already stressful, and adding a very slow backup experience was not helping.Restoring from backups was also too slow, both restoring from the backups themselves and replaying any WAL that had accrued since the last backup. (In Timescale, we automatically take full and incremental backups of all our customers’ databases.)In this blog post, we’re sharing how we solved this problem by combining pgBackRest with EBS snapshots. Timescale runs in AWS, so we had the advantage of cloud-native infrastructure. If you're running PostgreSQL in AWS, you can perhaps benefit from a similar approach.After introducing EBS snapshots, our backup creation and restore process got 100x faster.This significantly improved the experience for our customers and made things much easier for our team.Quick Introduction to Database Backups in PostgreSQL (And Why We Used pgBackRest)If you asked 100 engineers if they thought backups were important for production databases, they would all say ""yes""—but if you then took those same 100 engineers and gave them a grade on their backups, most wouldn’t hit a pass mark.We all collectively understand the need for backups, but it’s still hard to create an effective backup strategy, implement it, run it, and test that it’s working appropriately.In PostgreSQL specifically, there are two ways to implement backups:logical database dumps, which contain the SQL commands needed to recreate (not restore) your database from scratch, andphysical backups, which capture the files that store your database state.Physical backups are usually paired with a mechanism to store the constant stream of write-ahead logs (WALs), which describe all data mutations on the system. A physical backup can then be restored to get PostgreSQL to the exact same state as it was when that backup was taken, and the WAL files rolled forward to get to a specific point in time, maybe just before someone (accidentally?) dropped all your data or your disk ate itself.Logical backups are useful to recreate databases (potentially on other architectures), but maintaining physical backups is imperative for any production workload where uptime is valued. Physical backups are exact: they can be restored quickly and provide point-in-time recovery. In the rest of this article, we’ll discuss physical backups.How are physical backups usually created in PostgreSQL?The first option is using thepg_basebackupcommand.pg_basebackupcopies the data directory and optionally includes the WAL files, but it doesn’t support incremental backups and has limited parallelization capabilities. The whole process is very manual, too. If you’re usingpg_basebackup, you’ll instantly get the files you need to bootstrap a new database in a tarball or directory, but not much else.Tools likepgBackRestwere designed to overcome the limitations ofpg_basebackup. pgBackRest allows for full and incremental backups, multi-threaded operations, and point-in-time recovery. It ensures data integrity by validating checksums during the backup process, supports different types of storage, and much more. In other words, pgBackRest is a robust and feature-rich tool, making it our choice for PostgreSQL backup operations.The Problem With pgBackRestBut pgBackrest is not perfect: it reads and backs up files, causing an additional load on your system. This can cause performance bottlenecks that can complicate your backup and restore strategy, especially if you’re dealing with large databases.Even though pgBackRest offers incremental backups and parallelization, it often gets slow when executing full backups over large data volumes or on an I/O-saturated system.While you can sometimes rely on differential or incremental backups to minimize data (like we do in Timescale), there are situations in which creating full backups is unavoidable. Backups could also be taken on standby, but at the end of the day, you’re limited by how fast you can get data off your volumes.We shared earlier the example of full database upgrades, but we're also talking about any other kind of migration, integrity checks, archival operations, etc. In Timescale, some of our most popular platform features (likeforks,high-availability replicas, andread replicas) imply a data restore from a full backup.Having a long-running full backup operation in your production database is not only inconvenient, it can also conflict with other high-priority DB tasks, affecting your overall performance. This was problematic for us.The slowness of pgBackRest was also problematic when it was time to restore from these backups. It’s very good at CPU parallelization, but when you’re trying to write terabytes of data as fast as possible, I/O will be the bottleneck. When it comes to recovery time objective or RTO, every minute counts. In case of major failure, you want to get that database up as soon as possible.Using EBS Snapshots to Speed Up the Creation of BackupsTo speed up the process of creating fresh full backups, we decided to replace standard pgBackRest full backups with on-demandEBS snapshots.Our platform runs in AWS, which comes with some advantages. Using snapshots is a much more cloud-native approach to the problem of backups compared to what’s been traditionally used in PostgreSQL management.EBS snapshots create a point-in-time copy of a particular database: this snapshot can be restored, effectively making it a backup. The key is thattaking a snapshot is significantly faster than the traditional approach with pgBackRest: in our case, our p90 snapshot time decreased by over 100x. This gap gets wider the larger your database is!How did we implement this? Basically, we did a one-to-one replacement of pgBackRest. Instead of waiting for the pgBackRest fresh full backup to complete, we now take a snapshot. We still wait for the backup to complete, but the process is significantly faster via snapshots. This way, we get the quick snapshot but also the full data copy and checksumming for datafile integrity, which pgBackRest performs.If a user experiences a failure shortly after an upgrade, we have a fresh backup—the snapshot—that we can quickly restore (we’ll cover how we handle restores next). We still take a fresh full backup using pgBackRest (yay for redundancy), but the key difference is that this happens after the upgrade process has been fully completed.If a failure has happened, the service is available to our customer quickly: we don’t have to force them to wait for the lengthy pgBackRest process to finish before being able to use their service again.The trade-offs for adopting this approach were minimal. The only downside to consider is that, by taking snapshots, we now have redundant backups (both snapshots and full backups), so we incur additional storage costs. But what we’ve gained (both in terms of customer satisfaction and our own peace of mind) is worth the price.Combining EBS Snapshots and pgBackRest for Quick Data Restore: Taking Partial Snapshots, Replaying WALSolving the first problem we encountered with pgBackRest (i.e., slow creation of full backups) was relatively simple. We knew exactly when we needed an EBS snapshot to be created, as this process is always tied to a very specific workflow (e.g., performing a major version upgrade).But we also wanted to explore using EBS snapshots to improve our data restore functionality. As we mentioned earlier, some popular features in the Timescale platform rely heavily on restores, includingcreating forks,high-availability replicas, andread replicas,all of which imply a data restore from a full backup.This use case posed a slightly different and more difficult challenge since to restore from a full backup, such a backup needs to exist first, reflecting the latest state of the service.To implement this, the first option we explored was taking an EBS snapshot when the user clicked “Create” a fork, read replica, or high-availability replica, to then restore from that snapshot. However, this process was still too slow for the end user. To get the performance we wanted, we had to think a bit beyond the naive approach and determine a way to take semi-regular snapshots across our fleet.Fortunately, we already had a backup strategy for pgBackRest in place that we chose to mirror. Now, all Timescale services have EBS snapshots taken daily. For redundancy reasons and to verify file checksums, we still take our standard pgBackRest partial backups, but we don’t depend on them.Once the strategy is solved, restoring data from an EBS snapshot mirrors a restore from pgBackRest very closely. We simply chose the corresponding EBS snapshot we wanted to restore—in the cases mentioned above, always the most recent—and then replayed any WAL that has accumulated since that restore point. Here, it is important to note thatwe still rely on pgBackRest to do our WAL management. pgBackRest works great for us here; nothing gets close in terms of parallel WAL streaming.This EBS snapshotting and pgBackRest approach has given us great results so far. Using snapshots for restores has helped improve our product experience, also providing our customers with an even higher level of reliability. Keeping pgBackRest in parallel has given us peace of mind that we still have a traditional backup approach that validates our data as well as snapshots.We’re continually improving our strategy though, for example, by being smarter about when we snapshot—e.g., by looking at the accumulated WAL since the last snapshot to determine if we need to snapshot certain services more frequently. This practice helps improve restore times by reducing the amount of WAL that would need to be replayed, which is often the bottleneck in this process.On Snapshot PrewarmingOne important trade-off with this EBS snapshot approach is the balance between deployment time and initial performance. One limitation of a snapshot restore is that not all blocks are necessarily prewarmed andmay need to be fetched from S3the first time they are used, which is a slow process.To give props to pgBackRest restore, it does not have this issue. For our platform features, our trade-off was between getting the user a running read replica (or fork or high-availability replica) as quickly as possible or making sure it was as performant as possible.After some back and forth, we decided on our current approach on prewarming: we’re reading as much as we can for five minutes, prioritizing the most recently modified files first. The idea here is that we will warm the data the user is actively engaging with first. After five minutes, we then hand the process off to PostgreSQL to continue reading the rest of the volume at a slower pace until it is complete. For the initial warming, we use a customgoroutinethat reads concurrently from files.Backing It UpWe are not completely replacing our pgBackRest backup infrastructure with EBS snapshots anytime soon: it is hard to give up on the effectiveness and reliability of pgBackRest.But by combining EBS snapshots with pgBackRest across our infrastructure, we’ve been able to mitigate its performance problem significantly, speeding up our backup creation and restore process. This allows us to build a better product, providing a better experience to our customers.If you’re experiencing the same pains we were experiencing with pgBackRest, think about experimenting with something similar! It may cost you a little extra money, but it can be very much worth it.We still have work to do on our end: we will continue to iterate on the ideal snapshotting strategy across the fleet to minimize deployment times as much as possible. We are also looking at smarter ways to prewarm the snapshots and more applications for snapshots in general.If any of these problems interest you,check out our open engineering roles(we’re hiring!). And if you are a PostgreSQL user yourself,sign up for a free Timescale trialand experience the result of EBS snapshots in action.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/making-postgresql-backups-100x-faster-via-ebs-snapshots-and-pgbackrest/
2023-10-12T13:00:00.000Z,The Problem With Locks And PostgreSQL Partitioning (And How To Actually Fix It),"In my career, I have frequently worked for companies with large amounts oftime-partitioned data, where I was a software engineer focusing on our PostgreSQL databases.We’d already grown past the point where deleting data row-by-row was no longer practical, sowe needed to use PostgreSQL partitioning to manage data retention.In brief, dropping a whole partition allows PostgreSQL to remove the entire file from disk for a subset of your data rather than going through each row and removing them individually. So it’smuchfaster.But if you are doing partitioning natively in PostgreSQL, you do have to make sure to add new partitions where you’re ingesting new data and drop the old ones.This was a frequent cause of outages for us, even if we had reasonably well-tested scripts for adding and removing partitions. Unfortunately, the interactions around the scripts were less well-tested, and new, frequent and long-running queries prevented the partition management scripts from getting the locks required and creating new partitions. We didn’t see the problem at first because we’d created partitions a few days in advance, but then we ran out of time, and with no new partitions, we couldn’t insert, and whoops, down goes the app.These types of problems are particularly hard to debug and disentangle because they are often caused by totally unrelated pieces of code, in combination with changes in load. PostgreSQL has begun to address this with newer approaches attaching partitions concurrently, but they’re quite complex.I’ve seen the outages caused bypartitions failing to create, or disk filling up because they can’t be dropped, or the pauses in other, normal queries by partition management code. I know how difficult these problems can be. This is why TimescaleDB'shypertableswere so exciting to me when I discovered them, especially their lock minimization strategies.Understanding PostgreSQL LocksWhat are PostgreSQL locks?PostgreSQL locks are mechanisms that control concurrent access to data in the database to ensure consistency, integrity, and isolation of database transactions.PostgreSQL, like most other relational database management systems, is aconcurrentsystem, which means that multiple queries can be processed at the same time. Locks help in managing multiple transactions attempting to access the same data simultaneously, avoiding conflicts and potential data corruption.Why are PostgreSQL locks necessary?Concurrency is essential for optimizing the performance and responsiveness of the database. However, concurrency introduces several challenges that need careful handling to ensure the database’s integrity, consistency, and reliability:When multiple queries are executed concurrently, there's a risk that one transaction might view inconsistent or uncommitted data modified by another ongoing transaction. This can lead to erroneous results and inconsistencies in the database.Queries executing simultaneously can interfere with each other, leading to performance degradation, locking issues, or inconsistent data.When two transactions try to modify the same data simultaneously, it can lead to conflicts, data corruption, or loss of data.Locks are necessary to prevent these problems.Types of PostgreSQL locksPostgreSQL supports many different types of locks, but the three relevant to this article areACCESS SHARE,SHARE UPDATE EXCLUSIVE, andACCESS EXCLUSIVElocks.ACCESS SHARElocks are the least restrictive and are intended to prevent the database schema from changing under a query along with related caches being cleared. Access share locks are acquired for database read operations. The purpose of access share locks is to block access exclusive locks.SHARE UPDATE EXCLUSIVElocks allow concurrent writes to a table but block operations that change the database schema in ways that might interfere with running queries. These are used for some forms of concurrent schema changes in PostgreSQL, though two concurrent transactions cannot both take this lock on the same table. For example, you cannot concurrently detach and attach the same partition to/from the same parent table in different sessions. One must complete before the other starts. These locks generally are used for concurrency-safe schema changes, which do not clear cached relation information.ACCESS EXCLUSIVElocks are the most restrictive and are intended to prevent other queries from operating across a schema change. Access exclusive locks block all locks from all other transactions on the locked table.Cache invalidation andACCESS EXCLUSIVElocksFor performance reasons, PostgreSQL caches information about tables and views (which we call “relations”) and uses this cached information in query execution. This strategy is instrumental for PostgreSQL's efficiency, ensuring that data retrieval is quick and resource utilization is optimized.A critical scenario that needs meticulous handling lock-wise is when the structrure of tables is altered. When the schema is altered (e.g. by adding or dropping columns, changing data types, or modifying constraints) the cached information related to that table might become outdated or inconsistent. Therefore, it needs to be invalidated and refreshed to ensure that the query execution reflects the modified schema.To make this work, PostgreSQL takes an access exclusive lock on the table in question before the cached information for that relation can be invalidated.Using PostgreSQL Partitioning To Simplify Data ManagementIn PostgreSQLdeclarative partitioning, PostgreSQL tables are used both for empty parent tables and for partitions holding the data. Internally, each partition is a table, and there is mapping information used by the planner to indicate which partitions should be looked at for each query. This information is cached in the relation cache.When tables are partitioned based on time, it allows for an organized structure where data is segmented into specific time frames. This makes data management much faster, since dropping a whole partition allows PostgreSQL to remove the entire partition from disk rather than going through each row and removing them individually.In PostgreSQL, you can follow two general approaches for managing partitions and data retention, which as we'll see later, have two different concurrency considerations and problematics.Approach #1: Dropping partitionsIn the first approach, we simply drop partitions from a partitioned table when we want to delete data.CREATE TABLE partition_test (
    event_time timestamp,
    sensor_id bigint,
    reported_value float
) partition by range (event_time);

-- Create partition 
CREATE TABLE partition_test_2022 PARTITION OF partition_test 
FOR VALUES event_time FROM ('2022-01-01 00:00:00') TO ('2023-01-01 00:00:00');
```

--Drop partition
ALTER TABLE partition_test DROP PARTITION partition_test_2022;Approach #2: Concurrent workflowPostgreSQL also offers (in PostgreSQL 14 and newer) a concurrent workflow for these operations.CREATE TABLE partition_test_2022 (like partition_test);


ALTER TABLE partition_test ATTACH PARTITION partition_test_2022 FOR VALUES event_time FROM ('2022-01-01 00:00:00') TO ('2023-01-01 00:00:00') CONCURRENTLY;To remove a partition concurrently, we can:ALTER TABLE partition_test DETACH PARTITION partition_test_2022 CONCURRENTLY;


DROP TABLE partition_test_2022;The Problem With Locks And PostgreSQL PartitioningFrom a database administration perspective, neither of these approaches is very safe.Both the partition creation and dropping requires an access exclusive lock on thepartition_test, meaning that once the query is issued, no other queries can run against that table until the query is concluded and the transaction committed or rolled back. The locking in each case looks like this:In terms of the concurrent approach, it still has to address the issue of clearing the relation cache. It does so in two stages: first, a share update exclusive lock is takenpartition_test, and then information is written to the catalogs indicating that the table will be removed from the partition list. The backend then waits until all running queries have concluded (and all transactions guaranteeing repeatable reads have concluded) before removing the table from the partition map.This approach does not rely on locks to signal that the process is complete, only to prevent multiple concurrent updates for the status of the same set of partitions. As a result, even unrelated queries can block the detach operation. If the partition management script’s connection is interrupted for any reason, cleanup processes must be performed by the database administrator.Once the partition is removed from the partition list, it is locked in access exclusive mode and dropped. The locking approach of this process looks like this:In conclusion,The first approach (involving the manual creation and dropping of partitions) relatively quick operations but forces hard synchronization points on partitioned tables, which in time-series workloads are usually partitioned due to being heavily used. Problems here can cause database outages fairly quickly.The concurrent workflow doesn’t always solve these problems. In mixed-workflow applications, waiting for all running queries to complete (which can include long-running automatic maintenance tasks) can lead to long delays, dropped connections, and general difficulties in actually managing data retention. Particularly under load, these operations may not perform well enough to be useful.Common Advice on How to Fix This Problem (And Why It's Not The Best)The overall problems of partition management with time-series data fall into two categories:1) Failure to create partitions before they are needed can block inserts.2) Dropping partitions when needed for regulatory or cost reasons not only can fail but can also block reading and writing to the relevant tables.If you ask for advice, you'll probably hear one of these two things:Use custom scriptsMany companies begin their partition-management journey with custom scripts. This has the advantage of simplicity, but the disadvantage is that the operations can require heavy locks, and there is often a lack of initial knowledge on how to address these.Custom scripts are the most flexible approach to lock problems of partition management because of the entire toolkit (lock escalation, time-out and retry, and more). This allows knowledgeable teams to build solutions that work around the existing database workloads with the best success chance.On the other hand, this problem is full of general landmines, and teams often do not begin with the knowledge to navigate these hazards successfully.A second major problem with custom scripts is that database workloads can change over time, and this is often out of the hands of the responsible team. For example, a data science team might run workloads that interfere with production in ways the software engineering teams had not considered.Use pg_partmanpg_partmanprovides a general toolkit for partition management which can mitigate the problem on some workloads.pg_partmantakes a time-out-and-retry approach to partition creation and removal, meaning that—depending on the configuration and how things are run—the functions will run in an environment where a lock time-out is set. This prevents a failed lock from leading to an outage, but there is no guarantee that it will be obtained before the partitions are required.In most cases, you can tune these features to provide reasonable assurances that problems will usually be avoided. Workloads exist that prevent the partition management functions from successfully running in such an environment.pg_partmanis a good tool and an important contribution to this topic, but at scale and under load, it will only work in cases where you have a real opportunity to get the locks required within the lock time-out. I have personally worked in environments where important services would have to be briefly disabled to allow this to happen.How TimescaleDB Solves The Problem Of Locking in PostgreSQL PartitioningInstead of using PostgreSQL native partitioning, you caninstall the TimescaleDB extensionand usehypertables, which are PostgreSQL tables that are automatically partitioned. This solves the problems caused by locking, since hypertables minimize locks by design.TimescaleDB automatically partitions hypertables into chunks, organized by various partitioning criteria, usually time. This implementation is independent of PostgreSQL’s partitioning strategies and has been optimized as an independent add-on to PostgreSQL rather than a part of PostgreSQL core. TimescaleDB does not use inheritance as a table partitioning structure either, nor does TimescaleDB rely on the relation cache mentioned above for determining which chunks to scan.Within a TimescaleDB hypertable, chunks are added transparently as needed and removed asynchronously without intrusive locks on the parent table. TimescaleDB then uses various strategies to hook into the planner and execute TimescaleDB-specific approaches to partition selection and elimination. These strategies require locking the chunk table with intrusive locks but not locking the parent.This approach is likely to lead to some potential problems in serializable transaction isolation levels because once the underlying partition is gone, it is gone. In the event that a serializable transaction starts and then chunks are dropped, this will result in serialization errors or isolation violations.Lock minimizationPostgreSQL has traditionally taken the view that concurrency is not extremely important for database operations while Data Definition Language (DDL) commands are run. Traditionally, this is true. Even today, DDL commands are usually run sufficiently infrequently that the database cannot take the performance hit of introducing DDL commands as synchronization points.The emerging problems of heavy PostgreSQL users today are not usually performance problems but the fact that applications are often not written with an awareness of what these added synchronization points will mean. In my experience, these synchronization points themselves are a significant cause of database outages among large-scale PostgreSQL users.Timescale has been built to avoid the sort of locking problems that currently exist with PostgreSQL’s declarative partitioning simply because this is a common problem in time-series workloads.TimescaleDB maintains its own chunk catalogs and only locks the partitions that will be removed. The catalog entry is removed, then the chunk table is locked and dropped. Only an access share lock is taken on the top-level table. This means that reads and even writes can be done to other chunks without interfering with dropping or adding chunks.TimescaleDB’s current approach has one limitation when used under serializable transactions. Currently, if you use serializable transactions, there are certain circumstances where a transaction could go to read dropped chunks and no longer see them, resulting in a violation of the serialization guarantees. This is only a problem under very specific circumstances, but in this case, TimescaleDB behaves differently than PostgreSQL’s concurrent DDL approaches.In general, though, you should only drop chunks when you are reasonably sure they are not going to be accessed if you use serializable transaction isolation.Why is PostgreSQL not doing this?TimescaleDB’s solution cannot be perfectly replicated with stock PostgreSQL at the moment because dropping partitions requires active invalidation of cached data structures, which other concurrent queries might be using.Offering some sort of lazy invalidation infrastructure (via message queues, etc.) would go a long way to making some of this less painful, as would allowing more fine-grained invalidations to caching.ConclusionTimescaleDB’s approach to the problem of locking in the best solution today, better than the options available in stock PostgreSQL. But it's not yet perfect; it operates in between the two options given in terms of concurrency capabilities. We cannot drop a chunk that a serializable transaction has read until that transaction concludes regardless.Getting there is likely to require some changes to how the table and view characteristics are cached by PostgreSQL and how this cache invalidation works. I think that such improvements would help us on our way toward more transactional DDL, however.ManyALTER TABLEcommands are limited in concurrency largely because of these caching considerations. I think the general success of our approach here is also evidence of a need to address these limitations generally.In the meantime, if you're planning to partition your tables, check out Timescale. If you're running your PostgreSQL database in your own hardware,you can simply add the TimescaleDB extension. If you're running in AWS,try the Timescale platform for free.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-timescaledb-solves-common-postgresql-problems-in-database-operations-with-data-retention-management/
2023-09-01T20:18:00.000Z,Best Practices for Query Performance on PostgreSQL,"In aprevious blog post, we shared for the first time how we are building a distributed time-series database on PostgreSQL by relying on chunking instead of sharding. To that end, we also introduced distributed hypertables and showed how they could scale almost linearly for inserts up to a nine-node cluster (one access node and eight data nodes), with an insert rate of well over 12 million metrics a second.In this post, we take a look at how we optimize queries on distributed hypertables in our multi-node architecture and share some query benchmarks. We've seen significant performance benefits with distributed hypertables over regular hypertables for both full and partial aggregations.But first, let's look at the main causes of performance degradation on PostgreSQL.What Leads to Performance Problems?Performance issues can significantly impact the efficiency and functionality of a PostgreSQL database. Several factors can contribute to these problems. Here are some of the main causes:Inefficient Queries:One of the most common causes of performance problems is inefficient queries. Poorly optimized or complex queries can create bottlenecks and slow down database operations. Regularly reviewing and optimizing queries can help mitigate this issue.Insufficientindexes:A lack of appropriate indexes can lead to slower query execution as the database must scan entire tables to find relevant data.Implementing indexes on frequently queried columnscan significantly improve query performance.ImproperPostgreSQL datatypes:Using inappropriate data types can also impact performance. For example, using a text data type when an integer would suffice leads to unnecessary storage usage, potential casting overhead, and slower query speeds.Changing data volume:The query planner creates execution plans (how to run a query) based on statistics of source data. Increasing or decreasing the data volume can influence these stats and yield different execution plans, resulting in slow queries that were once fine. Analyzing the source table can help optimize the query for a different volume.Highvolume oftransactions:A high volume of transactions can strain system resources and lead to performance degradation. Implementing efficient transaction handling strategies can help manage this load.Insufficienthardwareresources:Hardware limitations, such as CPU, memory, or storage capacity, can hinder database performance. Upgrading hardware resources or optimizing resource allocation can often resolve these issues.Lock contention:Using locks on tables or rowscan slow down a system with parallel queries, basically synchronizing access to the data and creating waiting queues for requests. Locks should be used very carefully and only as a last result.Lack ofmaintenance:Routine maintenance tasks like vacuuming, reindexing, and updating statistics are crucial for maintaining optimal performance. Neglecting these tasks can lead to performance degradation over time.By addressing these common causes of performance issues, you can optimize your PostgreSQL database for better performance and responsiveness.Let's now jump straight into our benchmarks.Query Performance BenchmarksTo show the query performance of distributed hypertables, we use theTime-Series Benchmark Suite (TSBS)with the IT monitoring use case (DevOps). All of the benchmark results are produced on nodes that ran m5.2xlarge AWS instances.For the IT monitoring use case, we have a distributed hypertable with timestamped CPU metrics that cover multiple hosts being monitored. The distributed hypertable is partitioned along two dimensions:timeandhostname, where the hostname determines which data node a chunk is placed on.In other words, each data node stores data for the same time intervals but covers a specific subset of hosts, as shown below.Let’s first look at the query performance of computing the average hourly CPU usage per host.With eight data nodes, queries complete almost 8x faster than with one data node on a distributed hypertable.We include one data node as a point of reference to show the slight overhead of distributed hypertables due to the extra network communication and processing. Even with that slight overhead, we see that queries complete about 7x faster on a distributed hypertable with eight data nodes than on a regular (one node) hypertable.The reason this query performs well is because it can push most of the work down to the data nodes.Since the query is grouping based on hostname, each data node can compute afull aggregateindependently and concurrently (more on that later).But what if we query for something less ideal for this setup? For instance, let’s look at finding the max CPU usage per hour across a set of hosts. Such a query groups only on time, and data nodes can therefore only computepartial aggregates. To get the final result, the access node has to combine and finalize the partial results from the data nodes.Even in this case, eight data nodes are over 3x faster than a single data node.More notably, the overhead between hypertables and distributed hypertables is much smaller here than in the prior benchmark, bringing the two roughly on par. This is because the number of rows returned is much smaller for this query than the previous one, where we also grouped by hostname.How We Optimize Queries on Distributed Hypertables (With an Example)The key to unlocking the full query performance of distributed hypertables boils down to three main tactics:Limiting the amount of workOptimally distributing and pushing down work to data nodesKeeping data nodes busyWe’ll now discuss how Timescale incorporates each of these tactics.Limiting the amount of workOne of the first things that the access node does with a query is to determine which chunks are involved in the query and, by extension, which data nodes to talk to. For instance, let’s say we are executing the following query, which is similar to the one in the first benchmark test:SELECT time_bucket(time, ‘1 hour’) as hour, 
	hostname, avg(cpu) 
FROM measurements 
WHERE hostname IN (‘host001’, ‘host002’) 
	AND time > NOW() - interval ‘24 hours’
GROUP BY hour, hostname;This query tells us that we only want to work with the set of chunks whose time intervals overlap with the past 24 hours and include, e.g., hostnameshost001and/orhost002.Even though the access node doesn’t store any data locally, it has global knowledge of all chunks across the distributed hypertable, allowing it to performchunk exclusion. Each chunk in the resulting set has a list of data nodes that store a copy of it, allowing the access node to turn the set of chunks into a set of data nodes to be queried, as shown in the figure below.Note that chunks may have replica copies on multiple data nodes for fault-tolerance purposes; in that case, the access node has a choice of which data nodes to query for optimal performance.Different strategies for assigning chunks are possible, but the default strategy is always to use the designated “default” data node for a chunk.At the end of this planning stage, the access node has a list of data nodes to query, each with a disjoint subset of chunks. From this, it synthesizes a SQL query to send to each data node based on the original query:SELECT time_bucket(time, ‘1 hour’) as hour, 
	hostname, avg(cpu) 
FROM measurements 
WHERE _timescaledb_internal.chunks_in(measurement, ARRAY[1, 2])
hostname IN (‘host001’, ‘host002’) 
	AND time > NOW() - interval ‘24 hours’
GROUP BY hour, hostname;Note that the access node explicitly tells the data node which chunks to query via thechunks_infunction in theWHEREclause. This function serves two purposes: first, it obviates the need for running chunk exclusion again on the data node, and second, it avoids returning duplicate data from replica chunks that also exist on other data nodes.Thus, at the end of this planning stage, the access node knows exactly which data nodes to talk to and which chunks to query on those nodes.Pushing Down Work to Data NodesWhen generating the SQL query statement to send to a particular data node, the access node needs to decide which parts of the original query it can safely push down, i.e., execute on the data nodes, and which parts need to execute locally.In the worst case (i.e., no push down), the access node has to fetch the raw data from each data node and process it locally, as shown below:Clearly, fetching the raw data is to be avoided if possible, as it involves transferring a lot of data and puts a heavy processing burden on the access node. Instead, push-downs have the potential to improve the situation tremendously by (1) moving processing to the data nodes and (2) reducing the amount of transferred data since the output is often smaller than the input, especially in the case of aggregations and limits.Typical things the planner can consider to push down include:Functions (and general expressions)Sorting (ORDER BY)LIMITsAggregates and GROUP BYsWhile a number of things determine the ability to push down various parts of the query, we will focus on the ability to push down the computation of GROUP BY aggregates (e.g., calculating the average CPU usage per host and hour).Full aggregationGoing back to our example query, the first thing that determines the level of aggregate push-down is whether the GROUP BY clause covers all of the partitioning dimensions. Because the GROUP BY clause includes both time and hostname (both of which are partitioning dimensions), we know that it is safe to push down the aggregation fully.This is becauseno data for a time-hostname group on a data node can exist on any other data node. Thus, with full aggregate push-down, the query conceptually looks as follows:Note, however, that we aren’t actually grouping on time and hostname; we are grouping onhour(time bucket) and hostname. Fortunately, since we use hostname as our primary dimension along which we assign chunks to data nodes, the planner can think of this as a single-dimensional table partitioned only on hostname.This allows it to do time bucketing independently on each data node safely. There’s a caveat here, though: repartitioning. If the table has been repartitioned, data might not align along data node boundaries as we require, so full aggregation might not be possible (more on that below).In summary, full aggregation is possible if any of the following cases hold:The grouping clause includes all partitioning keysThe grouping clause includes only the “space” partitioning key, and the time restriction includes no repartitioning eventThe query is restricted to only one data nodeFortunately, the TimescaleDB planner is smart enough to detect and handle each case for optimal query performance.Partial aggregationPartial aggregation is necessary when the data for a computed group is not located on a single data node [for example, when we group only by hour (time)]. In this case, each data node computes a partial aggregate, and the access node then finalizes the result for each hour bucket.Conceptually, this looks as follows:While PostgreSQL fully supports partial aggregations on a local node (e.g., for parallel query execution), there is unfortunately no general way to express a partial aggregation in standard SQL, which is necessary to tell a data node that it should compute a partial aggregate.In the case ofavg(), this partial aggregate state would consist of a sum and a count that can be used to produce the final average. But, obviously, this state is different depending on the aggregate function. Fortunately, TimescaleDB has a function for computing the partial aggregate state that is also used bycontinuous aggregates.The SQL sent to a data node would then look something like this:SELECT time_bucket(time, ‘1 hour’) as hour, 
	_timescaledb_internal.partialize_agg(avg(cpu))
FROM measurements 
WHERE _timescaledb_internal.chunks_in(measurement, ARRAY[1, 2])
hostname IN (‘host001’, ‘host002’) 
	AND time > NOW() - interval ‘24 hours’
GROUP BY hour;Note the addition of thepartialize_aggfunction around the aggregate. This function tells the data node to compute and return the partial state for theavg()aggregate so that the access node can compute the final aggregate from all the data nodes' partial results.How repartitioning affects push-downIt is easy to expand the capacity of a distributed hypertable by adding additional data nodes. But to use the extra data nodes, existing distributed hypertables might require repartitioning to, e.g., increase the number of space partitions.However, since the new partitioning configuration only affects new chunks, the planner has to be careful to ensure that queries on data nodes still produce the correct results. To illustrate this situation, let’s look at what happens when we add a new data node to expand the capacity of a distributed hypertable:The figure shows that, during the third time interval, an extra data node was added so that the fourth time interval now includes four chunks instead of three. Now, imagine that the highlighted area shows the chunks covered by a query’s time and hostname restrictions.We find that this includesoverlappingchunks from two distinct partitioning configurations, i.e., data for a particular hostname might exist on more than one data node. This will prohibit full aggregations on data nodes, as it would otherwise produce the wrong result.Fortunately, TimescaleDB's planner can dynamically detect overlapping chunks and revert to the appropriate partial aggregation plan when necessary. Users can, therefore, freely add data nodes and repartition their data to achieve elasticity without worrying about the correctness of query results.While this leads to slightly worse query plans in some cases, these repartitioning events are rare and often quickly move out of the query or retention window. There’s also the possibility of rewriting old chunks in the new partitioning scheme, although no automation for such repartitioning of old data currently exists.Keeping data nodes busyTo minimize latency and maximize resource utilization, it is crucial that the access node keeps feeding data nodes with work. Unfortunately, the default behavior of PostgreSQL is to execute so-called Append plans serially, i.e., the access node starts with getting all tuples from the first data node, then moves on to the second, and so forth.This is obviously bad for performance and should be avoided. PostgreSQL 11 introduced parallel append plans, but they require launching separate worker processes on the access node, which is a lot of overhead when most of the work anyhow happens on data nodes. Further, parallel workers introduce other challenges related to read consistency and coordination of two-phase commit across multiple connections to the same data node.Instead of using parallel append, TimescaleDB introduces asynchronous append execution that allows the access node to asynchronously initiate the query work on each data node while fetching the results as they become ready.Basically, like the event-driven paradigm, the idea is to eliminate as much idle time as possible across both the access node and all data nodes. This provides great performance improvements for push-down aggregates since most of the work happens simultaneously on data nodes.Best Practices for Query PerformanceHere are some general practices that can further improve your query performance:Understand common performance bottlenecks:know the usual suspects of performance issues, such as inefficient queries, missing indexes, or hardware limitations.Utilize chunk exclusion:use TimescaleDB's chunk exclusionto optimize the selection of relevant data chunks for faster query processing.Implement full aggregate push-down:maximize the push-down of aggregation to data nodes when theGROUP BYclause covers all partitioning dimensions.Employ partial aggregation for complex queries:use partial aggregations when data for a computed group is spread across multiple data nodes.Monitor repartition effects on push-down:be aware of how repartitioning, especially with added data nodes, affects the optimization of push-down operations.Asynchronously execute appends:adopt asynchronous append execution to minimize idle time and initiate work on each data node concurrently.Continuously update and educate:regularly update your knowledge, tools,and PostgreSQL versionto benefit from performance improvements and stay informed on best practices.Next StepsIf you want to learn more about PostgreSQL performance issues, including how to identify and solve them, plus how to optimize your query performance, read the following resources:Using pg_stat_statements to Optimize QueriesPostgreSQL + TimescaleDB: 1,000x Faster Queries, 90 % Data Compression, and Much More13 Tips to Improve PostgreSQL Insert PerformanceTo try distributed hypertables, you canself-host TimescaleDBorsign up for a free 30-day trial of Timescale, our cloud solution, which includes our core database but supercharges it with features designed for higher performance, faster queries, and less spending.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/achieving-optimal-query-performance-with-a-distributed-time-series-database-on-postgresql/
2023-08-23T15:43:17.000Z,Best Practices for Picking PostgreSQL Data Types,"A good, future-proof data model is one of the most challenging problems when building applications. That is specifically true when working on applications meant to store (and analyze) massive amounts of data, such as time series, log data, or event-storing ones.Deciding what data types are best suited to store that kind of information comes down to a few factors, such as requirements on the precision of float-point values, the actual values content (such as text), compressibility, or query speed.In this installment of the best practices series (see our posts onnarrow, medium, and wide table layouts,single or partitioned hypertables, andmetadata tables), we’ll have a look at the different options in PostgreSQL and TimescaleDB regarding most of these questions. While unable to answer the requirement question, a few alternatives may be provided (such as integers instead of floating point)—but more on that later.Before We Start: CompressionEvent-like data, such as time series, logs, and similar use cases, are notorious for ever-growing amounts of collected information. Hence, it’ll grow continuously and require disk storage.But that’s not the only issue with big data. Querying, aggregating, and analyzing are some of the others. Reading this amount of data from disk requires a lot of I/O operations (IOPS; input/output operations per second), which is one of the most limiting factors in cloud environments, and even on-premise systems (due to how storage works in general). While non-volatile memory express (NVMes) transfer protocols and similar technologies can help you optimize for high IOPS, they’re not limitless.That’s where compression comes in. TimescaleDB’s compression algorithms (and, to some extent, the default PostgreSQL’s) help decrease disk space requirements and IOPS, improving cost, manageability, and query speed.But let’s get to the actual topic: best practices for data types in TimescaleDB.Basic Data TypesPostgreSQL (and the SQL standard in general) offers a great set of basic data types, providing a perfect choice for all general use cases. However, you should be discouraged from using them. ThePostgreSQL Wikiprovides a great list of best practices regarding data types to use or avoid. Anyhow, you don’t immediately have to jump over; we’ll cover most of them here 🥹.Nullable columnsWhen looking back at thebest practices on table layout, medium and wide table layouts tend to have a few too many nullable columns, being placeholders for potential values.Due to how PostgreSQL stores nullable values that areNULL, those are almost free. Having hundreds of nullable columns, most beingNULL, is not an issue. The same is true for TimescaleDB’s custom compression. Due to storing data in a columnar format, empty row values are almost free when compressed (null bitmap).Boolean valuesAbooleanvalue is a logical data type with one of two possible values,TRUEorFALSE. It is normally used to record decisions or states.There isn’t much specific to booleans in TimescaleDB. They are a very simple data type but also a great choice. Still, people often use anintegerto represent their values as1or0. This may come in handy with narrow or medium table layouts where you want to limit the number of columns. Nothing speaks against either solution!In terms of compressibility, booleans aren’t heavily optimized but compress fairly well with the standard compression. If you have a series of states, it may be recommended to only store state changes, though, removing duplicates from the dataset.Floating-point valuesFloating-point data types represent real numbers, most often decimal floating points of base ten. They are used to store all kinds of information, such as percentages, measurements like temperature or CPU usage, or statistical values.There are two groups of floating-point numbers in PostgreSQL,float4(a.k.a.,real),float8(double precision), andnumeric.Float4andfloat8columns are the recommended data types.TimescaleDB will handle them specifically (during compression) and optimize their use. On the other hand,numeric, as an arbitrary precision-sized data type, isn’t optimized at all.Numericisn’t recommended.In general, though, due to the complexity of floating-point numbers, if you know the required precision upfront, you could use the multiply-division trick and store them as integers, which are better optimized. For example, consider we want to store a temperature value (in Kelvin) and only two decimal places, but the value comes in as afloat4.float4 originalValue = 298.151566;
int storedValue = (round(originalValue * 100))::int;
float4 queryValue = storedValue::float4 / 100;It is a trick often used in data transmission for embedded devices with a low throughput uplink to limit the number of bytes sent.Integer valuesInteger data types represent natural numbers of various sizes (meaning valid number ranges, depending on how many bytes are used to represent them). Integer values are often used for simple values, such as counts of events or similar.All integer types (int2, SmallInt,int4, Integer,int8, BigInt) are recommended data types.TimescaleDB is heavily optimized to compress the values of those data types. No less than three compression algorithms are working in tandem to get the most out of these data types.This is why it is advised (if you know the necessary precision of a floating-point value) to store the values as integers (seefloating-point valuesfor more information).What is true for integers is also true for all serial data types (serial2, SmallSerial,serial4, Serial,serial8, BigSerial), as those are magical “aliases” for their integer counterparts, incorporating the automatic creation of sequences to fill in their values on insert.That said, they use their corresponding integer data types as a column data type. Anyhow, the PostgreSQL best practices advise against using them and recommend usingidentity columnsinstead for anything PostgreSQL from version 10 onwards.Timestamp, time, and date valuesTimestamps and time and date data types represent a specific point in time, some with more and some with less explicit information. All these data types have versions with and without timezone information attached (exceptdate).Before going into details, I’d generally advise against any of the data types without timezones (timestamp without time zone, timestamp, time without time zone, time).Most of them are discouraged by PostgreSQL’s best practices and shouldn’t be used.It’s a misconception that it would save any storage space, as many believe, and TimescaleDB doesn’t have any optimization for data types without timezones. While it works, it will add a lot of casting overhead which is implicit and, therefore, not immediately visible. That said, just don’t do it 🔥.With that out of the way, you can usedateandtime, but you should consider the use case. Whiledateis optimized and compressed using the same compression scheme as integers,timeis not. In any case, you shouldn’t use both for the time-dimension column.To store dates, you should consider usingtimestamptzwith the time portion set to midnight in the necessary timezone (2023-01-01T00:00:00+00) to prevent any casting overhead when querying.Likewise, you can usetimestamptzto store a time value only. In this case, you encode the time portion to a specific date (such as1970-01-01T15:01:44+04) and cast the final value back into atimevalue. Alternatively, you can store the value as anintegerby encoding the time into the (nano)seconds since midnight or any other encoding you can come up with.That leaves us withtimestamptz(timestamp with time zone). You’ve guessed it: this is the recommended data type for any kind of point-in-time storage.This data type is highly optimized, used by all internal functionality, and employs the same compression as integers. Said compression is especially effective with timestamps, as they tend to have little difference between two consecutive values and compress extremely well.Still, be aware that some frameworks, object-relational mappings (ORMs), or tools love theirtimestamp without time zoneand need to be forced to be good citizens.Text valuesText values are used to store textual values of arbitrary size. Those values can include detailed descriptions, log messages, and metric names or tags. The available data types includetext,char(n), andvarchar(n).PostgreSQL’s best practices advise against usingchar(n), as it will pad values shorter thannto that size and waste storage. It recommends usingtextinstead.The same is true withvarchar(n)with a length limit. Consider usingvarchar(without length limit) ortext.From a TimescaleDB-specific perspective, there isn’t much to say except you may want to deduplicate long values using a separate table holding the actual value and a reference (such as a checksum on the content) and storing the reference in the hypertable.TimescaleDB doesn’t offer any specific optimization to handle this type of data. It will, however, apply dictionary compression (lz4-based) to those text fields.Byte array (bytea) valuesByte arrays (in PostgreSQL represented by the data typebytea) store arbitrary large sequences of bytes, which may represent anything, from encoded machine state to binary data packets.When looking at customer/user use cases, it is a very uncommon data type, as most data is decoded before being stored in the database. Therefore, TimescaleDB doesn’t optimize anything about this data type. Compression, however, should use the lz4-based compression.If you have recurring, largebyteavalues, you can store them outside the actual hypertable and apply deduplication, as explained for text columns.Complex and Extension Data TypesCompared to basic data types, complex data types commonly encode multiple values into a single column. This may include custom composite types.Structural types (JSON, JSONB, XML)Structural data types encode complete objects or sets of information, oftenlossyin terms of data types of the actual values. PostgreSQL supports three structural data types,JSON,JSONB(a binary representation of JSON), andXML.Values often contain complex state information from machines or sensors. They are also common with narrow table layouts to compensate for the different values that need to be stored.To get it out of the way, if you want to store JSON-like data, don’t useJSON—useJSONB!It’s better regarding storage space, query speed, and anything you can think of.The only disadvantage ofJSONBis that you lose the original order of properties sinceJSONBwill decompose the object for better efficiency due to the way it is stored internally. Anyhow, not sure I can come up with a great reason for why the order should matter, and I hope you agree 😉.That said,JSONBis a valid choice when storing complex data.The amount of stored data stored should be kept under close observation, though. If you have recurring, large amounts of data inside theJSONBobjects, it may be advisable to extract that meta information into a separate, vanilla PostgreSQL table and join them in at query time. An approach is shown in thetext valuessection.For theXML,I don’t have enough data to give any recommendations. Actually, I cannot remember anyone ever asking anything about it. Due to my scarce experience, I wouldn't advise using it. I guess it may behave similarly toJSONB, but that’s reading clouds.For all the above data types, TimescaleDB won’t apply any specific optimizations, and compression will likely use the dictionary algorithm, which still yields impressive results (been there, done that).There are a few things to be wary of when using those data types. Remember that the whole object needs to be read, parsed, and the requested path’s value extracted. That happens for every single row of data being queried.One thing I learned using JSONB is thatindexes on expressionsandGIN indexesare your friends if you need to make selection decisions based on data inside the object (such as tags). Ensure that anything you need to make that decision is in an index to read and (potentially) decompress the object.The second element to remember is that extracting values from those objects yields text values that need to be cast into the required data type. Even if the value is stored as a number inside theJSONBobject, extracting it returns the text representation, which then needs to be cast back into an integer or floating-point value, adding a lot of casting overhead to the process.Universally Unique Identifiers: UUIDUsingUUIDto store unique identifiers is common in the PostgreSQL world. They are used for various reasons, including simplifying the generation of highly unlikely and colliding values on a different system than the database for “security” reasons (meaning removing potentially guessable series of numbers) and others.AUUIDrepresents a (semi)random 128-bit number in the format specified inRFC 4122(ISO/IEC 9834-8:2005), which looks similar toa0eebc99-9c0b-4ef8-bb6d-6bb9bd380a11.TimescaleDB doesn’t optimize this data type at all. It will compress it using the dictionary compression algorithm. However, there are other elements to keep in mind when usingUUID. While they offer many advantages and simplify distributed systems, they also create some locality issues, specifically with BTree indexes. You will find a great read on this topic in theCybertec blog.My recommendation would be to think twice aboutusingUUIDs.Tree-like structures: LtreeWhileltreeis an uncommon data type when storing event-like data, it can be a great fit for deduplicating recurring, larger values.Ltreerepresents a certain (sub)path in a tree, illustrating a specific device in an overarching topology. Their values resemble a canonical, dot-separated path (building1.floor3.room10.device12).Values of data typeltreearen’t handled by TimescaleDB specifically. However, due to their size, this isn’t a big deal.As mentioned above,ltreevalues are perfect for deduplicating recurring data, such as meta information, which are unlikely or slowly changing. Combined with a hash/checksum, they can be used as a reference to look them up in an external table.Key-value pairs:hstorehstoreprovides the capability to store key-value pairs of data, similar to what a non-nestedJSON(B)object would offer.From my experience, only a few people usehstore,and most preferJSONB. One reason may be thathstoreonly offers text values. That said, there is no experience in compression gains or speed implications. I guess a lot of what was noted onJSONBobjects would hold forhstore.PostGIS data: Geometries and geographiesLast but not least,PostGIS’data types are likegeometryandgeography. PostGIS data types are used to store special information of any kind. Typical use cases are GPS positions, street information, or positional changes over time.TimescaleDB works perfectly fine with PostGIS data typesbut doesn’t optimize them. Compression ratios, while not perfect, are still decent.Not a lot to mention here: they work, and they work great.What Is the Recommendation Now?Recommending data types is hard since many decisions depend on requirements outside the scope of this blog post. However, I hope this read provides insight into which data types to avoid, which ones are fine, and how you may optimize okay data types.To summarize, here’s a small table showing different aspects and categorizing them into four states: great 🟢, okay, but be wary of something 🟠, or avoid at any cost 🔴, as well as unknown (feel free to provide experience/feedback) ⚪️.Data TypeQuery SpeedCompressibilityRecommendedAlternativeboolean🟢🟢🟢real, float4🟢🟢🟢double (precision), float8🟢🟢🟢numeric🔴🔴🔴floats, integerssmallint, int2🟢🟢🟢integer, int4🟢🟢🟢bigint, int8🟢🟢🟢smallserial, serial2🟢🟢🟠int2 with identityserial, serial4🟢🟢🟠int4 with identitybigserial, serial8🟢🟢🟠int8 with identitytimestamp without time zone, timestamp🔴🟢🔴timestamptztimestamp with time zone, timestamptz🟢🟢🟢time🔴⚪️🔴timestamptztimetz🔴⚪️🔴timestamptzdate🔴🟢🔴timestamptzchar(n)🟢🟠🔴textvarchar(n)🟢🟠🔴texttext🟢🟠🟠bytea🟢🟠🟠json🔴🟠🔴jsonbjsonb🟢🟠🟠xml⚪️⚪️⚪️jsonbuuid🟠🟠🟠ltree🟢🟠🟢hstore⚪️⚪️⚪️jsonbgeometry, geography (PostGIS)🟢🟠🟢ConclusionIf you want to know more about the compression algorithms used, see the blog post one of my colleagues wrote,Time-Series Compression Algorithms, Explained.Also, if you have any feedback or experience with any of the above data types or the non-mentioned ones, I’d be happy to hear from you onTwitteror ourCommunity Slack(@Chris Engelbert)!Finally, I’m sorry for the wall of text you’ve had to read to get here, but there’s a lot of information to share and many different data types (and those aren’t even all of them). I hope you enjoyed the read and learned something new along the way.If you want to test Timescale right now, the easiest and fastest way to get started is to sign up for our30-day Timescale free trial. To try self-managed TimescaleDB, see thedocumentationfor further information.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/best-practices-for-picking-postgresql-data-types/
2023-10-24T14:12:07.000Z,"Database Backups and Disaster Recovery in PostgreSQL: Your Questions, Answered","When we ask our community about the elementary challenges they face with their PostgreSQL production databases, we often hear about three pain points: query speed, optimizing large tables, and managing database backups.To help you handle the first two, we’ve written about how to improve your PostgreSQL performance throughpartitioningandfine-tuning your database, and we’ve also discussed tactics onhow to reduce your database sizeto better manage large tables. Now, it’s time to talk about database backups!In this article, we’ll answer some of the most frequently asked questions we get on the topic of database backup and recovery in PostgreSQL, including how we handle things in theTimescale platform.Why Are PostgreSQL Database Backups Important?Let’s start from the beginning: why is everyone so worried about database backups?When we discuss backup and recovery, we’re referring to a set of processes and protocols established to safeguard your data from loss or corruption and restore it to a usable state:Backupsinvolve creating copies of your data at regular intervals, copies that encapsulate your PostgreSQL database state at a specific point in time.Recovery, on the other hand, is the process of restoring data from these backups. If both things are taken care of (i.e., you always have up-to-date backups and a good recovery strategy in place), your PostgreSQL database will be resilient against failure, and you’ll be protected against data loss.Effective backup management is not only about creating copies of data but also ensuring those copies are healthy, accurate, and up-to-date. To define a good backup strategy for your production PostgreSQL database, you need to consider a few aspects, including how frequently you will back up your database, where these backups will be stored, and how often you will audit them.But your job isn’t finished once you get up-to-date and healthy database backups: you must also establish an effective disaster recovery protocol. No matter how careful you are, it’s a fact of database management that, when running a production database, failures will happen sooner or later—whether outages cause them, failed upgrades, corrupted hardware, or human error—you name it.Your disaster recovery plan must encompass all the steps needed to restore data as quickly as possible after an incident, ensuring that your database is not just backed up but also recoverable in a timely and efficient manner.What Is the Difference Between a Physical Backup and a Logical Backup in PostgreSQL?In PostgreSQL, there are two main types of database backups: physical backups and logical backups.Physical backupscapture the database's state at a specific point in time. They involve copying the actual PostgreSQL database data at the file system level.Logical backupsinvolve exporting specific database objects or the entire database into a human-readable SQL file format. A logical backup contains SQL statements to recreate the database objects and insert data.Logical backups can be highly granular, allowing for the backup of specific database objects like tables, schemas, or databases. They are also highly portable and can be used across different database systems or versions, making them popular for migrating small to medium databases. This is your common pg_dump/pg_restore.But a main drawback of logical backups is speed. For large databases, the process of restoring from a logical backup is too slow to be useful as a sole disaster recovery mechanism (or migration mechanism, for that matter). Restoring from physical backups is faster than restoring from logical backups, and it’s exact; when putting together a disaster recovery strategy, you’ll be dealing with physical backups.Physical Backups in PostgreSQLLet’s explore some essential concepts around physical backups and how they can help you recover your database in case of failure.File system backupsFirst, to be technically exact, physical backups are referred to asfile system backupsin PostgreSQL. As mentioned earlier, this refers to the process of directly copying the directories and files that PostgreSQL uses to store its data, resulting in a complete representation of the database at a specific moment in time.Maintaining file system backups is an essential piece of every disaster recovery strategy and imperative in production databases. But putting together a solid disaster recovery plan requires other techniques beyond simply taking file system backups (or physical backups) regularly, especially if you’re dealing with large production databases.First, taking physical backups of very large databases can be a rather slow and resource-intensive process that can conflict with other high-priority database tasks, affecting your overall performance. Second, physical backups are not enough to ensure consistency in case of failure, as they only reflect the database state at the time they were taken. To restore a database in case of failure, you’ll need another mechanism to be able to restore all the transactions that occurred between the moment the last backup was taken and the failure.WAL and continuous archivingThat mechanism isWAL. WAL stands for Write-Ahead Logging, and it is a protocol that improves the reliability, consistency, and durability of a PostgreSQL database by logging changes before they are written to the actual database files.WAL is key for assuring atomicity and durability in PostgreSQL transactions. By writing changes to a log before they're committed to the database, WAL ensures that either all the changes related to a transaction are made or none at all.WAL is also essential for disaster recovery since, in the event of a failure, the WAL files can be replayed to bring the database back to a consistent state. The process of regularly saving and storing these WAL records in a secondary storage location, ensuring that they are preserved over the long term, is usually referred to as continuous archiving.Keeping WAL records and a recent, healthy physical database backup ensures that your PostgreSQL database can be successfully restored in case of failure. The physical backup will get PostgreSQL to the same state as it was when the backup was taken, which hopefully was not so long ago, and the WAL files will be rolled forward right before things start failing.You might be wondering why it’s necessary to keep up-to-date backups if WAL can be replayed. The answer is speed. Replaying WAL during a recovery process is time-consuming, especially when dealing with large datasets with complex transactions. Backups provide a snapshot of the database at a specific point in time, enabling quick restoration up to that point.In the optimal recovery scenario, a recent backup is restored (e.g., from the previous day), and then WALs recorded post-backup are replayed to update the database to its most recent state. You don’t want to rely on WAL to reproduce two weeks’ worth of transactions.What is point-in-time recovery (PITR) in PostgreSQL?Lastly, let’s define the concept of PITR. Point-in-time recovery refers to the ability to restore a PostgreSQL database to any specific point in time due to direct user input. For example, if I perform an upgrade and, for whatever reason, decide to revert the change, I could choose to recover the database from any day before.Behind the scenes, PITR in PostgreSQL is often anchored in WAL. By integrating a backup with the sequential replay of WAL, PostgreSQL can be restored to an exact moment.Which Tools Can I Use to Manage Physical Backups in PostgreSQL?Within the PostgreSQL ecosystem, there are multiple tools that help with the creation of physical backups, two of the most popular being pg_basebackup and pgBackRest.pg_basebackuppg_basebackupis the native tool offered by PostgreSQL for taking physical backups. It’s straightforward and reliable. It allows you to efficiently copy the data directory and include the WAL files to ensure a consistent and complete backup.pg_basebackup comes with important limitations. Taking full backups of a large database can be a lengthy and resource-intensive process. A good workaround to mitigate this is to combine full backups with incremental backups, i.e., frequently copying the data that has changed since the last full backup (e.g., once a day) and creating full backups less frequently (e.g., once a week). However, incremental backups are not supported in pg_basebackup.Pg_basebackup also has limited parallelization capabilities, which can slow down the creation of full backups even further. Additionally, the process is highly manual, requiring developers to closely monitor and manage the backup operations.pgBackRestTo address the constraints of pg_basebackup, tools likepgBackRestwere built by the PostgreSQL community. pgBackRest introduces several important improvements:It supports both full and incremental backups.It introduces multi-threaded operations, accelerating the backup process for larger databases.It validates checksums during the backup process to ensure data integrity, offering an additional layer of security.It supports various storage solutions, offering flexibility in how and where backups are stored.As we’ll explain in more detail in a later section, to manage our own backup and restore process in Timescale, we use pgBackRest, although we’ve implemented some hacks to speed up the process of executing full backups (pgBackRest can still be quite slow for creating backups in large databases).Logical Backups in PostgreSQLNow, let’s briefly cover logical backups. As we mentioned previously, logical backups involve exporting data into a human-readable format, such as SQL statements. This type of backup is generally more flexible and portable, making it handy to reproduce a database in another architecture (i.e., for migrations)—although recovering from a logical backup is quite a slow process, making them practical for migrating small to medium PostgreSQL production databases alone.pg_dump/pg_restoreThe most common way to create logical backups and restore from them is by using pg_dump/pg_restore:pg_dumpis used to create logical backups of a PostgreSQL database. It generates a script file or other formats that contain SQL statements needed to reconstruct the database to the state it was at the backup time. You can use pg_dump to back up an entire database or individual tables, schemas, or other database objects.pg_restoreis used to restore databases from backups created by pg_dump. Just as pg_dump offers granularity in creating backups, pg_restore allows for selective restoration of specific database objects, providing flexibility in the recovery process. While it is typically used with backups created by pg_dump, pg_restore is compatible with other SQL-compliant database systems, enhancing its utility as a migration tool.When Should I Use Logical Backups, and When Should I Use Physical Backups in PostgreSQL?Logical backups via pg_dump/pg_restore are mostly useful for creating testing databases or for database migrations.In terms of migrations, if you’re operating a production database, we only recommend going the pg_dump/pg_restore route if your database is small (<100GB).Migrating larger and more complex databases via pg_dump/pg_restore might take too much time, during which your production database would be offline. There are other migration strategies you can follow to avoid this downtime, like thedual-write and backfillmethod.Physical backups are mostly used for disaster recovery and for data archiving.If you are operating a production database, you’ll want to maintain up-to-date physical backups and WAL to recover your database when failure occurs. If your industry also requires you to keep copies of your data for a certain period of time due to regulations, physical backups will be the way to go.In production applications, you’ll most likely use a combination of both logical and physical backups. For disaster recovery, physical backups will be your foundational line of defense, but logical backups can serve as additional assurance (redundancy is a good thing). For migrating large databases, you’ll most likely use a staged approach,combining logical backups with other tactics, and so on.What About Replicas in PostgreSQL?We're not focusing specifically on replicas and high availabilityin this article, but these concepts are related to the topic of backups and disaster recovery, so let’s cover them.Replicas are continuously updated mirrors of the primary database, capturing every transaction and modification almost instantaneously. They're not the same as backups, but their usefulness in disaster recovery is indisputable: in the event of a failure, replicas can be promoted to serve as the primary database, ensuring minimal downtime while the damaged database is being restored.Building a high-availability replica and failover mechanism would generally involve taking care of the following steps:The primary database should be configured to allow connections from replicas.Physical backups of the primary should be regularly created, e.g., using pgBackRest.WAL capturing all changes made to the database should be shipped to the replica, for example, via streaming replication. Replication can be synchronous, where each transaction is confirmed only when both primary and replica have received it, or asynchronous, where transactions are confirmed without waiting for the replica.Configurations for automatic failover should be established to promote a replica to become the primary database in case of a failure.Tools and scripts should be used to monitor replication lag and ensure the replica is up-to-date.This setup can be considerably complex to maintain.Most providers of managed PostgreSQL databases, including Timescale, offer fully managed replicas as one of their services, making it much easier to run highly available databases.Database Backups and Disaster Recovery: How Do We Do It?TheTimescale platformallows our customers to create fully managed PostgreSQL and TimescaleDB databases, meaning that we take care of the backup and disaster recovery process for them. Let’s run through how the platform handles all things backups, replication, upgrades, and restores.How do backups work in Timescale?Backups in Timescale are fully automated. Using pgBackRest under the hood, Timescale automatically creates one full backup every week and incremental backups every day. Timescale also keeps WAL files of any changes made to the database. This WAL can be replayed in the event of a failure to reproduce any transactions not captured by the last daily backup, e.g., to replay the changes made to your database during the last few hours. The two most recent full backups and WAL are stored inS3volumes.On top of the full and incremental backups taken by pgBackRest,Timescale also takes EBS snapshots daily. EBS snapshots create copies of the storage volume that can be restored, effectively making it a backup, and they are significantly faster than taking full backups via pgBackRest (about 100x faster).By taking EBS snapshots daily (on top of the weekly full backups by pgBackRest), we introduce an extra layer of redundancy, ensuring that we always have a fresh snapshot that we can quickly restore if the customer experiences a critical failure that requires recovery from a full backup.If I’m a Timescale customer, do I have to keep my own backups?No. You can do it if you wish, but this would be (extra) redundant.Disaster recovery in Timescale: What happens if my database fails?Timescale is built on AWS with decoupled compute and storage, something that makes the platform especially resilient against failures. We can distinguish two types of failures that are dealt with differently by Timescale: compute and storage failures.How Timescale handles compute failuresCompute failures are more frequent than storage failures, as they can be caused by things like unoptimized queries or other issues that result in a maxed-out CPU. To improve uptime for the customer, Timescale has developed a methodology that makes the platform recover extremely quickly from compute failures: we call this techniquerapid recovery.Since the compute and storage nodes are decoupled in Timescale, if the compute node fails, Timescale automatically spins up a new compute node, attaching the undamaged storage unit to it. Any WAL that was in memory then replays. How long this recovery process takes mostly depends on how much WAL needs replaying, but it is usually completed in less than thirty seconds. Under the hood, this entire process is automated via Kubernetes.How Timescale handles storage failuresStorage failures are much less common than compute failures, but when they happen, they’re more severe. As we’ll cover in the next section, having ahigh-availability replicacan be a life-saver in this circumstance; while your storage is being restored, instead of experiencing downtime, your replica will automatically take over.To automatically restore your damaged storage, Timescale makes use of the backups it has on storage, reproducing WAL since the last incremental backup. The figure below illustrates the process:Recovery from backup in TimescaleHow do replicas work in Timescale, and how do they help with recovery?In Timescale, you can create two types of replicas:Read replicasare useful for read scaling. They’re used to liberate load from your primary database in read-heavy applications, for example, if you’re powering a BI tool or doing frequent reporting. Read replicas are read-only, and you can create as many as you need.High-availability replicasare exact, up-to-date copies of your database that automatically take over operations if your primary becomes unavailable.Let's dig deeper into high-availability replicas since they’re closer to the topic of backup and recovery discussed in this article.We’ve been talking about the importance of backups and disaster recovery, but there’s a related concept that’s important also to consider: the concept ofhigh availability. In broad terms, a “highly available” database describes a database that’s able to stay running without significant interruption (perhaps no more than a few seconds) even in case of failure.As we’ve mentioned previously, the process of recovering a large database from backup might take a while, even when you’ve done everything right—which is why it’s handy to have a replica running. Instead of waiting for the backup and restore process to finish, when your primary database fails, your connection will automatically failover to the replica, saving your own users any major downtime.Failover also helps remove downtime for common operations which would normally cause a service to reset, like some upgrades. In these cases, Timescale makes changes to each node sequentially so that there is always a node available.And speaking of upgrades…How are upgrades handled in Timescale?In Timescale, you’re running PostgreSQL databases with the TimescaleDB extension enabled. Therefore, during your Timescale experience, you’ll most likely experience three different types of upgrades:TimescaleDB upgradesThese refer to upgrades between TimescaleDB versions, e.g., from TimescaleDB 2.11 to TimescaleDB 2.12. You don’t have to worry about these: they’re backward compatible, they require no downtime, and they will happen automatically during your maintenance window. Your Timescale services always run the latest available TimescaleDB version, so you can enjoy all the new features we ship.PostgreSQL minor version upgradesWe always run the latest available minor version of PostgreSQL in Timescale as well, mostly for security reasons: these minor updates may contain security patches, data corruption problems, and fixes to frequent bugs.These upgrades are also automatically handled by the platform during your maintenance window, and they are also backward compatible. However, they require a service restart, which could cause some downtime (30 seconds to a few minutes) if you do not have a replica. We will alert you ahead of time about these, so you can set your maintenance window to a low traffic time (e.g., middle of the night) to minimize consequences.PostgreSQL major version upgradesThese refer to upgrading, for example, from PostgreSQL 15 to 16. These upgrades are different and more serious since they’re often not backward compatible. We cannot run these upgrades for you, as this might cause issues on your application. Besides, the downtime associated with upgrading major versions of PostgreSQL can be more severe (e.g., 20 minutes), and unfortunately, high-availability replicas can’t help you avoid downtime in this particular case.Major PostgreSQL upgrades are always a significant lift. Timescale has some tools that will make the smoother; for example,you can initiate the upgrade process in a particular database by clicking a button in the UI, and before doing so, you can test your upgrade in acopy of your databaseto make sure nothing will break and have an accurate idea of how much downtime the upgrade will require.Read this article for more information.Can I do PITR in Timescale, i.e., restore my database to a previous state at my own will?Yes you can!All Timescale services allow PITR to any point in the last 3 days. If you're using ourEnterprise plan, this timespan expands up to 14 days.The EndHaving a solid backup and recovery strategy is top of mind for every PostgreSQL user. We hope this introductory article answers some of your questions; if you’d like to see more articles diving deeper into this topic,tell us on Twitter/X.If you prefer not to worry about maintaining your backups and taking care of recovering your database when things fail,try Timescale, our managed PostgreSQL platform. It takes care of all things backups so you can focus on what matters (building and running your application) while experiencingthe performance boost of TimescaleDB. You can start a free trialhere, no credit card required.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/database-backups-and-disaster-recovery-in-postgresql-your-questions-answered/
2023-10-19T17:24:31.000Z,How to Use Psycopg2: The PostgreSQL Adapter for Python,"Python is one of the most popular programming languages, widely used for data analytics, visualizations, and data science. This guide will walk you through how to integrate PostgreSQL and your Python code via Psycopg2, one of the most popular PostgreSQL adapters.Connecting to a PostgreSQL Database Using PythonPostgreSQL adapters serve as bridges that enable you to directly interact with your PostgreSQL database directly from your application and programming language.In the particular case of Python, these are some of the most popular PostgreSQL adapters:pg8000pg8000stands out for its purity as a Python library and its seamless adherence to the Python Database API Specification v2.0. Unlike some adapters that rely on C extensions, pg8000 is written entirely in Python, which enhances its portability and ease of deployment across various environments.It's particularly favored for applications that need to avoid the complexities of dealing with C extensions while maintaining efficient communication with PostgreSQL databases. The adapter strikes a balance between simplicity and functionality, making it an excellent choice for developers who prioritize straightforward implementation and usage.pg8000 supports PostgreSQL 8.4 and up, as well as Python 2.6 to 2.7 and 3.2 to 3.7.asyncpgasyncpgis a distinct database adapter renowned for its superior performance and asynchronous processing capabilities. Designed explicitly forPython’s asyncio framework, asyncpg provides non-blocking, asynchronous communication with PostgreSQL databases. This ensures that applications remain responsive and scalable, particularly under heavy loads.Its specialization in handling concurrent database connections effectively distinguishes asyncpg from other adapters, making it a go-to option for developers building high-performance, I/O-bound applications.asyncpg supports PostgreSQL 9.2 and later versions, and it’s designed specifically for Python 3.5 and newer.SQLAlchemySQLAlchemyis not only an adapter but more of a comprehensive SQL toolkit and Object-Relational Mapping (ORM) system for Python applications. It abstracts the complexities of database communication, allowing developers to interact with databases using Pythonic expressions.SQLAlchemy's ORM enables developers to map Python objects to database tables, facilitating a higher-level, object-oriented perspective of database interaction. This feature-rich adapter is ideal for developers looking for an extensive set of tools to streamline both the basic and advanced aspects of database interaction.SQLAlchemy can be used with a variety of databases, including PostgreSQL.Psycopg2Lastly, the focus of this blog post. Psycopg2 is a popular adapter for its comprehensive feature set, robustness, and scalability, firmly establishing itself as a favorite among Python developers interfacing with PostgreSQL. It is implemented with C extensions, which contributes to its performance efficiency.Psycopg2 supports a range of PostgreSQL features, including server-side cursors, asynchronous notifications, and COPY commands. Furthermore, it is thread-safe and boasts connection pooling capabilities. Its widespread adoption is anchored on its reliability and compatibility with various versions of PostgreSQL and Python, making it a versatile choice for a diverse array of applications.psycopg2 supports PostgreSQL 7.4 and up and Python versions from 2.5 to 3.7.What Is the Difference Between Psycopg2 and SQLAlchemy?Psycopg2 and SQLAlchemy are very popular tools, so let’s spend a minute clarifying the difference between both.As we mentioned before, both tools are fundamentally different in character. SQLAlchemy is not only an adapter but an extensive SQL toolkit and ORM. With SQLAlchemy, developers can interact with databases using high-level Python expressions. It automatically translates these Python expressions into SQL code, reducing the need for writing SQL queries.This abstraction makes SQLAlchemy particularly beneficial for developers who are either less experienced with SQL or are looking for a more Pythonic way to interact with databases.In comparison, Psycopg2 offers a closer interaction with the PostgreSQL database, enabling developers to leverage PostgreSQL’s features to the fullest. It provides detailed control over database connections and query executions, making it a favorite for those who prioritize performance and direct database interaction.In sum, here are the main differences between Psycopg2 and SQLAlchemy:Psycopg2’s design is more straightforward, focusing on direct and efficient database interaction. SQLAlchemy, being an ORM, introduces an additional layer of abstraction, making database interactions more Pythonic and less complex.Psycopg2 is rich in features that allow for a closer and more intricate interaction with PostgreSQL databases. SQLAlchemy offers a broader set of tools that simplify not only database connections but also query executions, mapping Python objects to database tables, and other advanced functionalities.Psycopg2 provides developers with detailed control over SQL queries and database interactions. SQLAlchemy automates and abstracts many aspects of database communication, making it a suitable option for a less SQL-intensive experience.Using Psycopg2: Top AdvantagesNow, let’s get intoPsycopg2!This adapter seamlessly integrates Python and PostgreSQL, making it incredibly easy to work with these technologies in unison. It provides a set of Python modules that allow you to establish connections to PostgreSQL databases, execute SQL queries, and retrieve data easily. Psycopg2 adheres to Python’s database API specifications, ensuring a consistent and intuitive experience.These are its main strengths:Connection management.Psycopg2 excels in establishing and maintaining robust connections between Python applications and PostgreSQL databases. It guarantees a reliable and uninterrupted data transfer channel, enhancing the efficiency of data exchange and communication.Efficient SQL query execution.The adapter is equipped with capabilities for precise and rapid SQL query execution. It is adept at handling a variety of tasks, including data retrieval, record modifications, and executing complex operations, ensuring optimal performance and accuracy.Real-time data synchronization.Psycopg2 helps you develop real-time applications, as it ensures that Python code is consistently synchronized with the PostgreSQL database. This feature facilitates the creation of responsive, data-driven applications that can effectively adapt to dynamic data changes.Robusticity.Psycopg2 is recognized for its stability and reliability, making it the ideal choice for mission-critical applications. This library handles various PostgreSQL features, complex data types, and large objects, ensuring the precision and reliability needed for high-stakes projects.Cross-compatibility.Psycopg2 seamlessly integrates with different Python versions and PostgreSQL versions, providing cross-compatibility and versatility for your projects. This ensures that your Python code remains functional across various environments.Active community.Psycopg2 benefits from continuous development and a supportive community. This dedication to improvement ensures that Psycopg2 remains up-to-date with the latest advancements in Python and PostgreSQL, making it a reliable choice for your Python PostgreSQL interactions.Enhanced security.Psycopg2 prioritizes data security by offering robust features, including support for SSL connections. This added layer of security helps safeguard your sensitive data, maintaining data integrity and confidentiality in your transactions.When to Use Psycopg2: Example Use Cases of Using Python and PostgreSQLData analytics and reportingData analysts and scientists frequently employ Psycopg2 for seamless access to PostgreSQL databases. For instance, imagine a data analyst at a marketing firm who uses Psycopg2 to retrieve customer data from a PostgreSQL database. With this data, they can create insightful reports, analyze trends, and make data-driven decisions to enhance marketing strategies.Web developmentIn web development, Psycopg2 is invaluable for building dynamic, database-driven websites. Consider an e-commerce website where Psycopg2 is used to manage product inventory, customer orders, and user accounts stored in a PostgreSQL database. This ensures a smooth shopping experience for customers and efficient inventory management for the business.Business applicationsBusinesses across various industries leverage Psycopg2 for mission-critical applications. For example, a financial institution may employ Psycopg2 to maintain a secure and robust database of customer transactions and accounts. This ensures data integrity, reliability, and swift access to financial data.IoT and real-time applicationsIn the realm of IoT, Psycopg2 plays a crucial role in capturing and storing real-time sensor data. Imagine a smart city project that relies on Psycopg2 to collect and analyze data from various sensors, such as traffic cameras and air quality monitors. This data can be used to optimize traffic flow, improve air quality, and enhance overall city management.Scientific researchScientists and researchers utilize Psycopg2 for storing and analyzing scientific data. For instance, in a research project involving climate data, Psycopg2 could be used to store temperature and weather data in a Psycopg2 database. Researchers can then perform complex data analysis and generate climate models to better understand climate patterns.Installing Psycopg2: InstructionsBefore we begin, ensure that you have the following prerequisites in place:Python installed on your system (Python 3.6or later).Access to aPostgreSQLdatabase.Installation stepsPsycopg2 can be easily installed usingpip, Python's package manager. Open your terminal or command prompt and run the following command:pip install psycopg2You can also install the latest version ofPsycopg2, including the necessary binary dependencies using this command:pip install psycopg2-binary📚 Editor's note: If you're using Timescale,you can also find guidelines on how to install Psycopg2 in our documentation.To ensure that Psycopg2 is correctly installed, perform the following checks:Open a Python interpreter or create a Python script and import Psycopg2. If there are no import errors, Psycopg2 is installed correctly.You can verify the installed Psycopg2 version with the following code:print(psycopg2.__version__)Troubleshooting common installation issuesWhile Psycopg2 installation is generally straightforward, if you encounter some issues, run through this checklist:Psycopg2 relies on PostgreSQL’s C library. If you encounter missing library errors during installation, ensure you have the PostgreSQL development headers and libraries installed on your system.If you are using a virtual environment for your Python development, make sure it is activated when you run the installation command.Verify that your Python version meets Psycopg2's requirements. If you encounter compatibility issues, consider upgrading to a compatible Python version.Connecting to Your PostgreSQL Database Using Psycopg2Now that you have everything installed, let's connect to your PostgreSQL database!Here’s an example of establishing a database connection using Psycopg2:import psycopg2
# Defining database connection parameters
db_params = {
    ""host"": ""your_database_host"",
    ""database"": ""your_database_name"",
    ""user"": ""your_database_user"",
    ""password"": ""your_database_password"",
    ""port"": ""your_database_port""
}
try:
    # Establishing a connection to the database
    connection = psycopg2.connect(**db_params)
    # Creating a cursor object to interact with the database
    cursor = connection.cursor()
    # Performing database operations here...
except (Exception, psycopg2.Error) as error:
    print(f""Error connecting to the database: {error}"")

finally:
    if connection:
        cursor.close()
        connection.close()
        print(""Database connection closed."")In this code, we first import the psycopg2 module, which provides the functionality needed to interact with PostgreSQL databases from Python. Thedb_paramsdictionary contains the following parameters necessary to establish a database connection:host: specifies the hostname or IP address of the database server.database: specifies the name of the database you want to connect to.user: specifies the username for authentication.password: specifies the password for authentication.port: specifies the port number to connect to. The default is 5432 for PostgreSQL.The code is wrapped in a try block, which is used to handle exceptions or errors that may occur during the database connection process. Inside the try block,psycopg2.connect(**db_params)is used to establish a connection to the database. The**db_paramssyntax passes the connection parameters defined in the dictionary to the connect function.After successfully establishing a connection, a cursor object is created usingconnection.cursor(). The cursor is used to execute SQL queries and interact with the database. The except block catches any exceptions or errors that may occur during the connection process, and it prints an error message if there is an issue. Thefinallyblock ensures that the cursor and connection are properly closed.Running PostgreSQL Queries Using Psycopg2In this section, we assume that the database connection has already been established, so we’ll focus on showing some query examples you can run using Psycopg2.ExecutingSELECTqueries using Psycopg2The below code shows how to execute aSELECTquery with Psycopg2:# Defining the SELECT query
select_query = ""SELECT column1, column2 FROM your_table_name WHERE condition;""

# Executing the SELECT query
cursor.execute(select_query)

# Fetching and printing the results
result = cursor.fetchall()
for row in result:
    print(row)With this code, we achieve the following:We define theSELECTquery and include the desired columns and a condition.The execute method is used to execute the query.The results are fetched usingcursor.fetchall(), and we loop through the rows to print them.ExecutingINSERTqueries using psycopg2INSERT queries are executed similarly:# Defining the INSERT query
insert_query = ""INSERT INTO your_table_name (column1, column2) VALUES (value1, value2);""

# Executing the INSERT query
cursor.execute(insert_query)

# Committing the transaction to save changes
connection.commit()ExecutingUPDATEandDELETEqueries using Psycopg2Same with UPDATES and DELETES:# Defining the UPDATE query
update_query = ""UPDATE your_table_name SET column1 = new_value WHERE condition;""

# Executing the UPDATE query
cursor.execute(update_query)

# Committing the transaction to save changes
connection.commit()# Defining the DELETE query
delete_query = ""DELETE FROM your_table_name WHERE condition;""

# Executing the DELETE query
cursor.execute(delete_query)

# Committing the transaction to save changes
connection.commit()Troubleshooting: Fixing Common Psycopg2 ErrorsNow that you have everything up and running, let’s explore the most frequently encountered errors you might see when working with Psycopg2 to interact with PostgreSQL databases, guiding you on how to remedy them.psycopg2.OperationalErrorimport psycopg2

try:
    connection = psycopg2.connect(
        dbname=""yourdbname"",
        user=""youruser"",
        password=""yourpassword"",
        host=""yourhost"",
        port=""yourport""
    )
    # Additional database operations here

except psycopg2.OperationalError as error:
    print(f""OperationalError: {error}"")Thepsycopg2.OperationalErroris one of the common errors that you might see. It generally encapsulates one of these issues:Connection failure.The problem might be related to establishing a connection to the PostgreSQL server. Ensure that the server is up and running and accessible from the client machine.Invalid database name or credentials.Mistyping or incorrect credentials also can lead to anOperationalError. Make sure that the database name, username, and password are correct.Networking issues.Networking issues, such as incorrect host, port, or network unreachable, can also trigger this error. Make sure that the networking configurations are correct and that the server is reachable.Insufficient privileges.The user might not have the required privileges to connect to the specified database. Ensure the user has the necessary permissions.Server overload or downtime.The PostgreSQL server might be down or experiencing overload issues—confirm that the server is in good health and operational.In order to fix the error, you can go through these steps. You’ll typically resolve the issue:Ensure that the PostgreSQL server is running and listening on the correct port.Confirm that the connection parameters (including host, port, user, password, and database name) are correct.Ensure the network connection between the client and server is stable and that firewalls or network ACLs are not blocking the connection.Confirm that the user has the required permissions to connect to the database.Examine the PostgreSQL and system logs for additional details on the error.psycopg2.ProgrammingErrorimport psycopg2

try:
    connection = psycopg2.connect(
        dbname=""yourdbname"",
        user=""youruser"",
        password=""yourpassword"",
        host=""yourhost"",
        port=""yourport""
    )
    cursor = connection.cursor()
    # Replace the next line with your actual SQL query
    cursor.execute(""YOUR SQL QUERY HERE"")
    connection.commit()

except psycopg2.ProgrammingError as error:
    print(f""ProgrammingError: {error}"")This error is usually a signal of an anomaly within the structure or syntax of the SQL query being executed. These are the most common reasons:SQL syntax errors.You might have a typos, misplaced operator, or other mistake in your SQL syntax.Invalid table/column names.You might be referencing non-existent tables or columns.Incorrect data types.This error also shows up if you have mismatched data types or if you’re attempting to insert an incorrect data type into a column.Privilege issues. Lastly, you might be trying to access or manipulate the database objects without the necessary privileges.To troubleshoot this error, run through this checklist:Review the SQL query for any syntax errors. UtilizeSQL lintersor built-in database functions to check the syntax.Ensure that all referenced table and column names exist and are spelled correctly.Ensure that you are inserting data that matches the expected data types of the columns.Verify if the database user has adequate privileges to execute the intended operations on the database, tables, or columns.Also, pay close attention to the error message—it often contains information on what’s wrong with the SQL query.psycopg2.IntegrityErrorimport psycopg2

try:
    connection = psycopg2.connect(
        dbname=""yourdbname"",
        user=""youruser"",
        password=""yourpassword"",
        host=""yourhost"",
        port=""yourport""
    )
    cursor = connection.cursor()
    # Replace the next line with your actual SQL query
    cursor.execute(""YOUR SQL QUERY HERE"")
    connection.commit()

except psycopg2.IntegrityError as error:
    print(f""IntegrityError: {error}"")
    connection.rollback()This error occurs when an attempted operation threatens database integrity constraints, for example:Unique constraints violation (i.e., attempting to insert a duplicate value in a column that is constrained to have only unique values).Foreign key constraints violation (i.e., trying to insert a value in a foreign key column that does not exist in the referenced primary key column).Check constraints violation (i.e., inserting data that does not satisfy the check constraints of the columns).Not null constraints violation (i.e., attempting to insert a null value into a column that is defined asNOT NULL).Here's how you can troubleshoot:Review the error message, as it usually contains specific information about the nature of the integrity violation.Ensure that the data being inserted adheres to the integrity constraints defined for the table.Review the integrity constraints on the table to understand the rules and ensure compliance.psycopg2.DataErrorimport psycopg2

try:
    connection = psycopg2.connect(
        dbname=""yourdbname"",
        user=""youruser"",
        password=""yourpassword"",
        host=""yourhost"",
        port=""yourport""
    )
    cursor = connection.cursor()
    # Replace the next line with your actual SQL query
    cursor.execute(""YOUR SQL QUERY HERE"")
    connection.commit()

except psycopg2.DataError as error:
    print(f""DataError: {error}"")
    connection.rollback()This error is related to data values, specifically when the type or format of the data being inserted or manipulated is not compatible with the expected data type of the database column. To fix it, examine the data being inserted or updated to ensure its type, format, and value are compatible with the column’s specifications.Psycopg2 Best PracticesIn addition to the troubleshooting tips we shared in the previous section, some general best practices will help you run Psycopg2 with fewer errors (and fix them quicker when they arise).Logging errorsBy implementing a logging mechanism in your Psycopg2 workflow, it’ll be easier to track and analyze complex issues. This is a great practice for applications that interact with databases.For example, consider the following code:import psycopg2
import logging

# Configuring logging
logging.basicConfig(filename='database_errors.log', level=logging.ERROR, 
                    format='%(asctime)s:%(levelname)s:%(message)s')

try:
    connection = psycopg2.connect(
        dbname=""yourdbname"",
        user=""youruser"",
        password=""yourpassword"",
        host=""yourhost"",
        port=""yourport""
    )
    cursor = connection.cursor()
    cursor.execute(""YOUR SQL QUERY HERE"")
    connection.commit()

except (Exception, psycopg2.Error) as error:
    # Rolling back the transaction in case of error
    connection.rollback()
    # Logging the error
    logging.error(f""Error: {error}"")
    print(f""Error: {error}"")

finally:
    # Closing the cursor and connection
    if connection:
        cursor.close()
        connection.close()This logging configuration is enriched with a format parameter to include a timestamp, severity level, and the error message. This allows you to record your errors more comprehensively, ensuring that transactions are rolled back in the event of an error and avoiding partial data commits that can lead to inconsistencies.It's also essential to close database resources like cursors and connections to prevent resource leakage, and this can be effectively handled within a final block.Always check your database connectionVerify that the database connection is still open before executing queries to avoid errors related to a closed connection:import psycopg2

# Assuming 'connection' is the established database connection

try:
    # Checking if the connection is still open
    if connection.closed == 0:
        # Database operations here
    else:
        print(""Database connection is closed."")
except (Exception, psycopg2.Error) as error:
    print(f""Error: {error}"")Sanitize your input to prevent SQL injection attacksA SQL injection attack is a type of security vulnerability that occurs when an attacker is able to insert malicious SQL code into a query. This can happen when an application allows user input to be included in SQL queries without proper validation or escaping.When the database executes this malicious input, it can lead to unauthorized access, data theft, corruption, or other adverse impacts. This is something you want to protect yourself from when working with Psycopg2.In this snippet, user-provided input is sanitized and validated to prevent SQL injection attacks. Thesafe_inputvariable is created using Psycopg2'sadaptfunction, ensuring that user input is properly escaped and can be safely used in SQL queries:import psycopg2

# User-provided input
user_input = ""'; DROP TABLE users --""

try:
    # Sanitizing and validating user input
    safe_input = psycopg2.extensions.adapt(user_input).getquoted()
    
    # Using safe_input in our query
    query = f""SELECT * FROM users WHERE username = {safe_input};""

    # Executing the query
    cursor.execute(query)

except (Exception, psycopg2.Error) as error:
    print(f""Error: {error}"")Using Psycopg2 and TimescaleDBIn this section, we will cover a few examples of TimescaleDB operations that can be effectively managed through Psycopg2.TimescaleDBis a PostgreSQL extension that boosts PostgreSQL performance fordata-intensive applications dealing with time-series data, such as IoT sensor data, energy metrics, tick financial data, and many more.Creating hypertables using Psycopg2Hypertablesare the core feature of TimescaleDB. Theyautomatically partition data by time, improving query and ingest performance and making data management more efficient.To transform your PostgreSQL tables into hypertables using Psycopg2, run:import psycopg2

# Assuming a table 'your_table' with a time column 'time_column'
cursor.execute(""SELECT create_hypertable('your_table', 'time_column');"")Creating continuous aggregates using Psycopg2Continuous aggregatesare another of the most loved TimescaleDB features. This feature significantly improves query performance by calculating and storing frequently used aggregates for faster retrieval. It is based on PostgreSQL materialized views, adding improvements like automatic updates and progressive refreshes.import psycopg2

# Creating a continuous aggregation for 'your_hypertable'
cursor.execute(""SELECT create_continuous_aggregate('your_continuous_agg', 'your_hypertable');"")Querying TimescaleDB using Psycopg2You can query TimescaleDB databases seamlessly using Psycopg2, just as you query PostgreSQL. You can leveragethe library of SQL functionsthat TimescaleDB offers to write time-based queries more effectively—for example, using functions like time_bucket, which allows you to group entries into specified time intervals, simplifying the process of aggregating and analyzing data over time.For example, in this SQL query, the time_bucket function is used to create time intervals of one hour, facilitating the grouping and aggregation of temperature data within these intervals. The query then calculates the average temperature for each location within each time bucket, offering insights into temperature trends over time and across locations.SELECT 
    time_bucket('1 hour', time) as one_hour_interval,
    location,
    AVG(temperature) as avg_temperature
FROM 
    conditions 
WHERE 
    temperature > 76 
GROUP BY 
    one_hour_interval, location 
ORDER BY 
    one_hour_interval DESC, avg_temperature DESC;ConclusionIn this comprehensive guide, we've delved into the world ofPsycopg2and how it can help you connect to your PostgreSQL database from your Python application.If you could do a query performance boost in your PostgreSQL database, giveTimescalea try. This PostgreSQL extensionwill make your queries faster viaautomatic partitioning, query planner enhancements, improved materialized views,columnar compression, and much more.If you're running your PostgreSQL database on your own hardware,you can simply add the TimescaleDB extension. If you prefer to try Timescale in AWS,create a free account on our platform. It only takes a couple of seconds, no credit card required.Next stepsAlso, don't forget to check out some of our Python resources, from time-series data analysis to OpenAI exploration:How to Work With Time Series in Python?Tools for Working With Time-Series Analysis in PythonPostgreSQL vs Python for Data Cleaning: A GuideDo More on AWS With Timescale: Build an Application Using Lambda Functions in PythonJupyter Notebook Tutorial: Setting Up Python & Jupyter Notebooks on macOS for OpenAI ExplorationIngest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-use-psycopg2-the-postgresql-adapter-for-python/
2023-07-26T13:00:22.000Z,Supercharge Your AI Agents With Postgres: An Experiment With OpenAI's GPT-4,"Hello developers, AI enthusiasts, and everyone eager to push the boundaries of what's possible with technology! Today, we're exploringAI agentsas intermediaries in a fascinating intersection of fields: Artificial Intelligence and databases.The Dawn of AI AgentsAI agents are at the heart of the tech industry's ongoing revolution. As programs capable of autonomous actions in their environment, AI agents analyze, make decisions, and execute actions that drive a myriad of applications. From autonomous vehicles and voice assistants to recommendation systems and customer service bots, AI agents are changing the way we interact with technology.But what if we could take it a step further? What if we could use AI to simplify how we interact with databases? Could AI agents act as intermediaries, interpreting human language and converting it into structured database queries?A Ruby Experiment With GPT-4That's exactly what we tried to achieve in a recent experiment. Leveraging OpenAI's GPT-4, a powerful language model, we conducted an experiment to see how we could use AI to interact with our databases using everyday language.The experiment was built using Ruby, and you can find thedetailed explanation and code here. The results were fascinating, revealing the potential power of using AI as a “middle-man” (Middle-tech? Middle-bot?) between humans and databases.Check out the videos throughout this blog postto see it in action:Why Store Data for AI Agents?Data storage is crucial for the successful application of AI, particularly for training and fine-tuning models. By storing interactions, results, and other relevant data, we can improve the performance and accuracy of our AI agents over time.But data storage is not just about improving our AI; it's also about cost-effectiveness. With the OpenAI API, you pay per token, which can add up when dealing with large amounts of data. By using PostgreSQL as long-term memory for your AI agent, you can reduce the number of tokens you send to the OpenAI API, saving computational resources and money.PostgreSQL: Flexible and RobustPostgreSQL is a powerful, open-source relational database system. With a reputation for reliability, robustness, and performance, it's a fantastic choice for your AI's long-term memory. PostgreSQL also offers flexibility and scalability, making it suitable for projects of all sizes.Whether you're conducting experiments or deploying production-ready applications, PostgreSQL's flexibility and robust nature make it an excellent companion for your AI.Needless to say, we’re huge PostgreSQL enthusiasts here at Timescale—so much so that we built Timescale on PostgreSQL. Timescale works just like PostgreSQL under the hood, offering the same 100 percent SQL support (not SQL-like) and a rich ecosystem of connectors and tools but supercharging PostgreSQL for analytics, events, and time series (and time-series-like workloads).With additional features likecompressionandautomatically updated incremental materialized views—we call them continuous aggregates—Timescale allows you to scale PostgreSQL further for optimal performance while enjoying the best developer experience and cost-effectiveness.But why all this talk about Timescale? As the conversation between human and machine is happening on point in time, I realize I’m dealing with time-series data. Cue in TimescaleDB for the rescue!Join the Timescale CommunityWe're just scratching the surface of what's possible when combining AI with databases like PostgreSQL, and we'd love for you to join us on this journey.Got a cool idea? A question? Or just want to share your thoughts on this topic? Join the Timescale Community onSlackand head over to the#ai-llm-discussionchannel. Let's push the boundaries together and shape the future of AI!Check this page to learn how topower agents, chatbots, and other large language models AI applications with PostgreSQL. To see what my fellow Timescalers Avthar, Mat, and Sam are already building, read their post onPostgreSQL as a Vector Database: Create, Store, and Query OpenAI Embeddings With pgvector.Remember, technology grows exponentially when great minds come together. See you there!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/supercharge-your-ai-agent-with-postgresql-an-experiment-with-openais-gpt-4/
2023-10-25T18:48:16.000Z,What Is TOAST (and Why It Isn’t Enough for Data Compression in Postgres),"If you’re working with large databases in Postgres, this story will sound familiar. As your Postgres database keeps growing, your performance starts to decline, and you begin to worry about storage space—or, to be precise, how much you’ll pay for it. You love PostgreSQL, but there’s something you wish you had: a highly effective data compression mechanism.PostgreSQL does have somewhat of a compression mechanism:TOAST🍞. In this post, we’ll walk you through how Postgres TOAST works and the different TOASTing strategies.As much as we enjoy a good TOAST, we’ll discuss why this is not the kind of compression feature you need for reducing the storage footprint of modern large databases—and how, as the PostgreSQL enthusiasts that we are here at Timescale, we decided to build a more suitable compression mechanism for PostgreSQL, inspired by the columnar design of NoSQL databases.What Is Postgres TOAST?Even if it might reduce the size of datasets, TOAST (The Oversized Attribute Storage Technique) is not your traditional data compression mechanism. To understand what TOAST is, we have to start by talking abouthow data is stored in PostgreSQL.Postgres’ units of storage are called pages, and these have a fixed size (8 kB by default). Having a fixed page size gives Postgres many advantages, namely itsdata management simplicity, efficiency, and consistency, but it comes with a downside: some data values might not fit within that page.This is where TOAST comes in. TOAST refers to the automatic mechanism that PostgreSQL uses to efficiently store and manage values in Postgres that do not fit within a page. To handle such values, Postgres TOAST will, by default, compress them using an internal algorithm. If, after compression, the values are still too large, Postgres will move them to a separate table (called the TOAST table), leaving pointers in the original table.(As we’ll see later in this article, you can actually modify this strategy as a user, for example, by telling Postgres to avoid compressing data in a particular column.)TOAST-able Data TypesThe data types that might be subjected to TOAST are primarily variable-length ones that have the potential to exceed the size limits of a standard PostgreSQL page. On the other hand, fixed-length data types, likeinteger,float, ortimestamp, are not subjected to TOAST since they fit comfortably within a page.Some examples of data types that might be subjected to TOAST are:jsonandjsonbLargetextstringsvarcharandvarchar(n)(If the length specified invarchar(n)is small enough, then values of that column might always stay below the TOAST threshold.)byteastoring binary dataGeometric data likepathandpolygonand PostGIS types likegeometryorgeographyHow Does Postgres TOAST Work?Understanding TOAST not only relates to the concept of page size but also to another Postgres storage concept: tuples. Tuples are rows in a PostgreSQL table. Typically, the TOAST mechanism kicks in if all fields within a tuple have a total size of over 2 kB approx.If you’ve been paying attention, you might be wondering, “Wait, but the page size is around 8 kB—why this overhead?” That’s because PostgreSQL likes to ensure it can store multiple tuples on a single page: if tuples are too large, fewer tuples fit on each page, leading to increased I/O operations and reduced performance.Postgres also needs to keep free space to fit additional operational data: each page stores not just the tuple data but also additional information for managing the data, such as item identifiers, headers, and transaction information.So, when the combined size of all fields in a tuple exceeds approximately 2 kB (or the TOAST threshold parameter, as we’ll see later), PostgreSQL takes action to ensure that the data is stored efficiently. TOAST handles this in two primary ways:Compression.PostgreSQL can compress the large field values within the tuple to reduce their size using a compression algorithm that we’ll cover later in this article. By default, if compression is sufficient to bring the tuple's total size below the threshold, the data will remain in the main table, albeit in a compressed format.Out-of-line storage.If compression alone isn't effective enough to reduce the size of the large field values, Postgres moves them to a separate TOAST table. This process is known as ""out-of-line"" storage because the original tuple in the main table doesn’t hold the large field values anymore. Instead, it contains a ""pointer"" or reference to the location of the large data in the TOAST table.(We’re simplifying things slightly for the purpose of this article—read the PostgreSQL documentation for a full detailed view.)The Postgres Compression Algorithm:pglzWe’ve mentioned that TOAST can compress large values in PostgreSQL. But which compression algorithm is PostgreSQL using, and how effective is it?Thepglz(PostgreSQL Lempel-Ziv) is the default internal compression algorithm used by PostgreSQL specifically tailored for TOAST. Here’s how it works in very simple terms:pglztries to avoid repeated data. When it sees repeated data, instead of writing the same thing again, it just points back to where it wrote it before. This ""avoiding repetition"" helps in saving space.Aspglzreads through data, it remembers a bit of the recent data it has seen. This recent memory is referred to as the ""sliding window.""As new data comes in,pglzchecks if it has seen this data recently (within its sliding window). If yes, it writes a short reference instead of repeating the data.If the data is new or not repeated enough times to make a reference shorter than the actual data,pglzjust writes it down as it is.When it's time to read the compressed data,pglzuses its references to fetch the original data. This process is quite direct, as it looks up the referred data and places it where it belongs.pglzdoesn't need separate storage for its memory (the sliding window); it builds it on the go while compressing and does the same when decompressing.This implementation is designed to offer a balance between compression efficiency and speed within the TOAST mechanism. In terms of compression rate, the effectiveness ofpglzwill largely depend on the nature of the data.For example, highly repetitive data will compress much better than high entropy data (like random data). You might see compression ratios in the range of 25 to 50 percent, but this is a very general estimate—results will vary widely based on the exact nature of the data.Configuring TOASTTOAST strategiesBy default, PostgreSQL will go through the TOAST mechanism according to the procedure explained earlier (compression first and out-of-line storage next, if compression is not enough). Still, there might be scenarios where you might want to fine-tune this behavior on a per-column basis. PostgreSQL allows you to do this by using the TOAST strategiesPLAIN,EXTERNAL,EXTENDED, andMAIN.EXTENDED:This is the default strategy. It implies the data will be stored out of line in a separate TOAST table if it’s too large for a regular table page. Before moving the data to the TOAST table, it will be compressed to save space.EXTERNAL:This strategy tells PostgreSQL to store the data for this column out of line if the data is too large to fit in a regular table page, and we’re asking PostgreSQL not to compress the data—the value will just be moved to the TOAST table as-is.MAIN:This strategy is a middle ground. It tries to keep data in line in the main table through compression; if the data is definetely too large, it'll move the data to the TOAST table to avoid an error, but PostgreSQL won't move the compressed data. Insead, it’ll store the value in the TOAST table in its original form.PLAIN:UsingPLAINin a column tells PostgreSQL to always store the column's data in line in the main table, ensuring it isn't moved to an out-of-line TOAST table. Take into account that if the data grows beyond the page size, theINSERTwill fail because the data won’t fit.If you want to inspect the current strategies of a particular table, you can run the following:\d+ your_table_nameYou'll get an output like this:=> \d+ example_table
                     Table ""public.example_table""
 Column  |       Data Type   | Modifiers | Storage  | Stats target | Description 
---------+------------------+-----------+----------+--------------+-------------
  bar    | varchar(100000)  |           | extended |              | 
  ```If you wish to modify the storage setting, you can do so using the following command:-- Sets EXTENDED as the TOAST strategy for bar_column 
ALTER TABLE example_blob ALTER COLUMN bar_column SET STORAGE EXTENDED;Key parametersApart from the strategies above, these two parameters are also important to control TOAST behavior:TOAST_TUPLE_THRESHOLDThis is the parameter that sets the size threshold for when TOASTing operations (compression and out-of-line storage) are considered for oversized tuples.As we’ve mentioned previously, by default,TOAST_TUPLE_THRESHOLDis set to approximately 2 kB.TOAST_COMPRESSION_THRESHOLDThis is the parameter that specifies the minimum size of a value before Postgres considers compressing it during the TOASTing process.If a value surpasses this threshold, PostgreSQL will attempt to compress it. However, just because a value is above the compression threshold, it doesn't automatically mean it will be compressed: the TOAST strategies will guide PostgreSQL on how to handle the data based on whether it was compressed and its resultant size relative to the tuple and page limits, as we’ll see in the next section.Bringing it all togetherTOAST_TUPLE_THRESHOLDis the trigger point. When the size of a tuple's data fields combined exceeds this threshold, PostgreSQL will evaluate how to manage it based on the set TOAST strategy for its columns, considering compression and out-of-line storage. The exact actions taken will also depend on whether column data surpasses theTOAST_COMPRESSION_THRESHOLD:EXTENDED(default strategy):If a tuple's size exceedsTOAST_TUPLE_THRESHOLD, PostgreSQL will first attempt to compress the oversized columns if they also exceedTOAST_COMPRESSION_THRESHOLD. If compression brings the tuple size under the threshold, it will remain in the main table. If it doesn't, the data will be moved to an out-of-line TOAST table, and the main table will contain pointers to this external data.MAIN:If the tuple size surpasses theTOAST_TUPLE_THRESHOLD, PostgreSQL will try to compress the oversized columns (provided they're over theTOAST_COMPRESSION_THRESHOLD). If compression allows the tuple to fit within the main table's tuple, it remains there. If not, the data is moved to the TOAST table in its uncompressed form.EXTERNAL: PostgreSQL skips compression, regardless of theTOAST_COMPRESSION_THRESHOLD. If the tuple's size is beyond theTOAST_TUPLE_THRESHOLD, the oversized columns will be stored out-of-line in the TOAST table.PLAIN: Data is always stored in the main table. If a tuple's size exceeds the page size (due to having very large columns), an error is raised.StrategyCompress if tuple > TOAST_COMPRESSION_THRESHOLDStore out-of-line if tuple > TOAST_TUPLE_THRESHOLDDescriptionEXTENDEDYesYesDefault strategy. Compresses first, then checks if out-of-line storage is needed.MAINYesOnly in uncompressed formCompresses first, and if still oversized, moves to TOAST table without compression.EXTERNALNoYesAlways moves to TOAST if oversized, without compression.PLAINNoNoData always stays in the main table. If a tuple exceeds the page size, an error occurs.Why TOAST Isn't Enough as a Data Compression Mechanism in PostgreSQLBy now, you’ll probably understand why TOAST is not the data compression mechanism you wish you had in PostgreSQL. Modern applications imply large volumes of data ingested daily, meaning databases (over)grow quickly.Such a problem was not as prominent when our beloved Postgres was built decades ago, but today’s developers need compression solutions for reducing the storage footprint of their datasets.While TOAST incorporates compression as one of its techniques, it's crucial to understand that its primary role isn't to serve as a database compression mechanism in the traditional sense. TOAST is mainly a solution to one problem: managing large values within the structural confines of a Postgres page.While this approach can lead to some storage space savings due to the compression of specific large values, its primary purpose is not to optimize storage space across the board.For example, if you have a 5 TB database made up of small tuples, TOAST won’t help you turn those 5 TB into 1 TB. While there are parameters within TOAST that can be adjusted, this won't transform TOAST into a generalized storage-saving solution.And there are other inherent problems with using TOAST as a traditional compression mechanism in PostgreSQL, for example:Accessing TOASTed data can add overhead, especially when the data is stored out of line. This becomes more evident when many large text or other TOAST-able data types are frequently accessed.TOAST lacks a high-level, user-friendly mechanism for dictating compression policies. It’s not built to optimize storage costs or facilitate storage management.TOAST's compression is not designed to provide especially high compression ratios. It only uses one algorithm  (pglz) with compression rates varying typically from 25-50 percent.Adding Columnar Compression to PostgreSQL With TimescaleVia the TimescaleDB extension, PostgreSQL users have a better alternative. Inspired by the compression design of NoSQL databases,we added columnar compression functionality to PostgreSQL. This transformative approach transcends PostgreSQL’s conventional row-based storage paradigm, introducing the efficiency and performance of columnar storage.By adding a compression policy to your large tables,you can reduce your PostgreSQL database size by up to 10x (achieving +90 percent compression rates).By defining a time-based compression policy, you indicate when data should be compressed. For instance, you might choose to compress data older than seven (7) days automatically:-- Compress data older than 7 days
SELECT add_compression_policy('my_hypertable', INTERVAL '7 days');Via this compression policy, Timescale will transform the tablepartitions(which in Timescale arealso created automatically) into a columnar format behind the scenes, combining many rows (1,000) into an array. To boost compressibility,  Timescale will apply different compression algorithms depending on the data type:Gorilla compressionfor floatsDelta-of-delta +Simple-8bwithrun-length encodingcompression for timestamps and other integer-like typesWhole-row dictionary compression for columns with a few repeating values (+ LZ compression on top)LZ-based array compression for all other typesThis columnar compression design offers an efficient and scalable solution to the problem of large datasets in PostgreSQL. It allows you to use less storage to store more data without hurting your query performance (it actually improves it).And in the latest versions of TimescaleDB, you can alsoINSERT,DELETE, andUPDATEdirectly over compressed data.Keep ReadingHave we piqued your curiosity? Read the following blog posts to learn more about compression in Timescale:Building Columnar Compression in a Row-Oriented DatabaseHow Ndustrial Is Providing Fast Real-Time Queries and Safely Storing Client Data With 97 % CompressionAllowing DML Operations in Highly Compressed Time-Series Data in PostgreSQLTime-Series Compression Algorithms, ExplainedWrap-UpWe hope this article helped you understand that while TOAST is a well-thought-out mechanism to manage large values within a PostgreSQL page, it’s not effective for optimizing database storage use within the realm of modern applications.If you’re looking for effective data compression that can move the needle on your storage savings, give Timescale a go. You can try our cloud platformthat propels PostgreSQL to new performance heights, making it faster and fiercer—it’s free, and no credit card is required—or you can addthe TimescaleDB extensionto your self-hosted PostgreSQL database.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/what-is-toast-and-why-it-isnt-enough-for-data-compression-in-postgres/
2023-09-05T13:00:58.000Z,A Deep Dive Into PostgreSQL Vacuum Monitoring With BPFtrace,"Observability enables engineers to quickly troubleshoot and fix issues in production. For those using the Linux kernel, theextended Berkeley Packet Filter (eBPF) technologyallows application monitoring with minimal overhead. For example, you can useUProbesto trace the invocation and exit of functions in programs.Modern database observability tools (e.g.,funccount) are built on top of eBPF. However, these fully flagged tools are often written in C and Python and require some development effort when a ""quick and dirty"" solution for a particular observation is sometimes more than sufficient. However, withBPFtrace, a high-level tracing language for Linux eBPF, users can create eBPF programs with only a few lines of code.In this tutorial, we develop a simple BPFtrace program to observe the execution of vacuum calls in PostgreSQL and measure and print their execution times.The EnvironmentPostgreSQLusesvacuum operationsto reclaim space from dead (e.g., updated or deleted) tuples.For more information on how this process works and why it is an important one to monitor, read this article.In this tutorial, we will trace the vacuum calls and determine the required time per table for the vacuum operations. We will use a PostgreSQL 14 server, with the  binary is located at/home/jan/postgresqlsandbox/bin/REL_14_2_DEBUG/bin/postgres.In addition, the examples are executed in a database with these two tables:CREATE TABLE testtable1 (
   id int NOT NULL,
   value int NOT NULL
);

CREATE TABLE testtable2 (
   id int NOT NULL,
   value int NOT NULL
);❗Note:Depending on the used C compiler and applied optimizations, the symbols of internal functions (i.e., as static declared) may not be visible. In this case, you can not use UProbes to trace the function invocations. There are two possible solutions to address this issue: (1) remove the static modifier from the function declaration and recompile PostgreSQL, or (2) create a completedebug buildof PostgreSQL.Tracing PostgreSQL Vacuum Operations UsingfunclatencyLet’s explore the existing solutions before developing our tool to trace the vacuum operations.The toolfunclatencyis available for most Linux distributions (on Debian, it’s part of the packagebpfcc-tools and renamed to funclatency-bpfcc) and allows it to trace a function enter and exit and measure the function latency (i.e., the time it takes a function to complete).In PostgreSQL, the functionvacuum_relis invoked when a vacuum operation on a relation is performed. To trace these function calls withfunclatency-bpfcc, you need to provide the path of the PostgreSQL binary and the function name. For instance:‌$ sudo funclatency-bpfcc -r /home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel

Tracing 1 functions for ""/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel""... Hit Ctrl-C to end.Afterward, an eBPF program is loaded into the Linux kernel, and a UProbe is defined and observes the beginning of the function call while another UProbe observes its exit. The latency between these two events is measured and stored.To execute some vacuum operations, we perform the following SQL statement in a second session:database=# VACUUM FULL;
VACUUM FULL‌This SQL statement triggers PostgreSQL to perform a vacuum operation of all tables of the current database—this takes some time. After the vacuum operations are done, thefunclatency-bpfccprogram can be stopped (by executing CTRL+C), ending the observation of the binary and showing the recorded execution times on the terminal.$ sudo funclatency-bpfcc -r /home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
[...]
^C
Function = b'vacuum_rel' [876997]
     nsecs               : count     distribution
         0 -> 1          : 0        |                                        |
         2 -> 3          : 0        |                                        |
         4 -> 7          : 0        |                                        |
         8 -> 15         : 0        |                                        |
        16 -> 31         : 0        |                                        |
        32 -> 63         : 0        |                                        |
        64 -> 127        : 0        |                                        |
       128 -> 255        : 0        |                                        |
       256 -> 511        : 0        |                                        |
       512 -> 1023       : 0        |                                        |
      1024 -> 2047       : 0        |                                        |
      2048 -> 4095       : 0        |                                        |
      4096 -> 8191       : 0        |                                        |
      8192 -> 16383      : 0        |                                        |
     16384 -> 32767      : 0        |                                        |
     32768 -> 65535      : 0        |                                        |
     65536 -> 131071     : 0        |                                        |
    131072 -> 262143     : 0        |                                        |
    262144 -> 524287     : 0        |                                        |
    524288 -> 1048575    : 0        |                                        |
   1048576 -> 2097151    : 0        |                                        |
   2097152 -> 4194303    : 0        |                                        |
   4194304 -> 8388607    : 2        |*                                       |
   8388608 -> 16777215   : 13       |***********                             |
  16777216 -> 33554431   : 44       |****************************************|
  33554432 -> 67108863   : 7        |******                                  |
  67108864 -> 134217727  : 1        |                                        |

avg = 22765358 nsecs, total: 1525279002 nsecs, count: 67

Detaching...‌The output contains the information that the functionvacuum_relwas called 67 times, and the average function time is22765358 nsecs.In addition, a histogram of the function latency is printed, providing a lot of helpful information. However, while we now know the number and duration of vacuum calls, we still don’t know the duration of the individual vacuum calls for each relation.This is something that this tool does not support because it does not evaluate the function parameters (e.g., the Oid of relation that the current function invocation should vacuum). But this is something we can do withbpftrace.Using BPFtrace To Trace VACUUM FULL EntriesLet’s start with a very simple BPFtrace program that prints a line once thevacuum_relfunction is invoked in the PostgreSQL binary.bpftraceis called with the eBPF program that should be loaded into the Linux kernel. The eBPF programs that are passed to BPFtrace have the followingsyntax:<probe1> {
        <Actions>
}

[...]

<probeN> {
        <Actions>
}‌The syntax to define a UProbe on a binary or library is:uprobe:library_name:function_name[+offset]. For instance, to define a UProbe on the function invocation ofvacuum_relin the binary/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgresand print the lineVacuum started, you can use the following BPFtrace call:$ sudo bpftrace -e '
uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel {
    printf(""Vacuum started\n"");
}
'

Attaching 1 probe...
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
Vacuum started
[...]As soon as theVACUUM FULLSQL statement in PostgreSQL is executed in another terminal session, the program starts to print the message on the screen. This is a good start, but we still have less information available than output by the existing toolfunclatency-bpfcc. We're missing the latency of the function calls.Tracing Vacuum Function Returns and LatencyTo measure the execution time of the function invocations, we need two things:Define a second Uprobe that is invoked when the function observed returns; this can be done by using a URetProbe.Calculate the difference between the two UProbe events.A URetProbe in BPFtrace can be defined using the same syntaxuretprobe:binary:functionas the one used for the UProbe. In addition, BPFtrace allows it to create variables like associative arrays.We use such an array to capture the start time of a function invocation@start[tid] = nsecs;. The array's key is the ID of the current thread:tid. So, multiple threads (and processes like in our case with PostgreSQL) can be traced simultaneously without overriding the last function invitation start time.In the URetProbe we take the current time and subtract the time of the function invocation (nsecs - @start[tid])) to get the time the function call needs. We also use a function predicate (/@start[tid]/) to let BPFtrace know that we only want to execute the function body of the URetProbe as soon as this array value is defined.Using this predicate, we prevent handling a function return without seeing the function enter before (e.g., we start theBPFtrace program in the middle of a running function call, and we get only the URetProbe invocation for this function call).❗Note:It is not guaranteed that BPFtrace will deliver and process the eBPF events in order. Especially when a function call is short, and we have a lot of function invocations, the events could be processed out-of-order (e.g., we see two function enter events followed by two function return events). In this case, function latency observations with BPFtrace become imprecise. To avoid this, we use VACUUM FULL calls instead of vacuum calls. These calls aremuch more expensivesince they rewrite the table. Therefore, they take longer and can be reliably observed by BPFtrace.$ sudo bpftrace -e '
uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf(""Performing vacuum\n"");
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf(""Vacuum call took %d ns\n"", nsecs - @start[tid]);
        delete(@start[tid]);
}
'After running this BPFtrace call and executingVACUUM FULLin a second session, we see the following output:Attaching 2 probes...
Performing vacuum
Vacuum call took 37486735 ns
Performing vacuum
Vacuum call took 16491130 ns
Performing vacuum
Vacuum call took 32443568 ns
Performing vacuum
Vacuum call took 17959933 ns
[...]‌For each call of thevacuum_relfunction in PostgreSQL, we measure the time the vacuum operation needs. However, it would also be convenient to capture the Oid or the name of the relation vacuumed by the current vacuum operation. This requires the handling of the function parameters of the observed function.Handle Function ParametersThe functionvacuum_relhas the following signature in PostgreSQL 14. The first parameter is theOid(anunsigned int) of the processed relation. The second parameter is aRangeVarstruct, whichcouldcontain the name of the relation. The third parameter is aVacuumParamsstruct, which contains additional parameters for the vacuum operation, and the last parameter is aBufferAccessStrategy, which defines the access strategy of the used buffer.static bool vacuum_rel(Oid relid,
        RangeVar *relation,
        VacuumParams *params,
        BufferAccessStrategy bstrategy 
)BPFtrace allows it to access the function parameter using the keywordsarg0,arg1, …,argN. To include the Oid in the output of our logging, we need only to print the first parameter of the function.$ sudo bpftrace -e '

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf(""Performing vacuum of Oid %d\n"", arg0);
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf(""Vacuum call took %d ns\n"", nsecs - @start[tid]);
        delete(@start[tid]);
}
'When theVACUUM FULLoperation is executed again in a second terminal, the output looks as follows:Attaching 2 probes...
[...]
Performing vacuum of Oid 1153888
Vacuum call took 37486734 ns
Performing vacuum of Oid 1153891
Vacuum call took 49535256 ns
Performing vacuum of Oid 2619
Vacuum call took 39575635 ns
Performing vacuum of Oid 2840
Vacuum call took 40683526 ns
Performing vacuum of Oid 1247
Vacuum call took 14683600 ns
Performing vacuum of Oid 4171
Vacuum call took 20587503 ns‌To determine which Oid belongs to which relation, the following SQL statement can be executed:blog=# SELECT oid, relname FROM pg_class WHERE oid IN (1153888, 1153891);
   oid   |  relname   
---------+------------
 1153888 | testtable1
 1153891 | testtable2
(2 rows)The result shows that Oids1153888and1153891belong to the tablestesttable1andtesttable2, which we have created in one of the first sections of this article. These values belong to our test environment. In your environment, different OIDs might be shown.Handle Function Struct ParametersSo far, we have processed simple parameters withbpftrace(like Oids, which are unsigned integers). However, many parameters in PostgreSQL are C data structs. Furthermore, these structs can be handled in BPFtrace programs as well.The second parameter of thevacuum_relfunction is a RangeVar struct. This struct isdefined in PostgreSQL 14as follows:typedef struct RangeVar
{
	NodeTag	type;
	char *catalogname;
	char *schemaname;
	char *relname;
	[...]
}To process the struct, the following BPFtrace program can be used. Please note, that the internalNodeTagdata type of PostgreSQL is replaced by a simple int.TheNodeTagdata type is anenum. Enums are backed by the integer data type in C. To handle this enum correctly, we could (1) also copy the enum definition into the eBPF program, or (2) we could replace it with a data type of the same length.To keep the BPFtrace program simple, the second option is used here. The next three struct members are char pointer, which contains thecatalogname, the schema, and the name of the relation. Theschemanameand therelnameare the fields we are interested in. The struct contains more members, but these members are ignored to keep the example clear.$ sudo bpftrace -e '
struct RangeVar
{
	int type;
	char *catalogname;
	char *schemaname;
	char *relname;
};

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        printf(""[PID %d] Performing vacuum of Oid %d (%s.%s)\n"", pid, arg0, str(((struct RangeVar*) arg1)->schemaname), str(((struct RangeVar*) arg1)->relname));
        @start[tid] = nsecs;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{
        printf(""[PID %d] Vacuum call took %d ns\n"", pid, nsecs - @start[tid]);
        delete(@start[tid]);
}
'After the struct is defined, the members of the struct can be accessed as in a regular C program. For example:((struct RangeVar*) arg1)->schemaname. In addition, we also print the process ID (PID) of the program that has triggered the UProbe. This allows it to identify the process that has performed the vacuum operation.When running the following SQL statements in a second terminal:VACUUM FULL public.testtable1;
VACUUM FULL public.testtable2;The BPFtrace program shows the following output:Attaching 2 probes...
[PID 616516] Performing vacuum of Oid 1153888 (public.testtable1)
[PID 616516] Vacuum call took 23683600 ns
[PID 616516] Performing vacuum of Oid 1153891 (public.testtable2)
[PID 616516] Vacuum call took 24240837 nsThe table names are extracted from theRangeVardata structure and shown in the output. However, this data structure is not always populated by PostgreSQL. The data structure might be empty when runningVACUUM FULLwithout specifying a table name. Therefore, we use two single invocations with explicit table names to force PostgreSQL to populate this data structure.Optimizing the BPFtrace Program Using MapsThe BPFtrace programs we have developed so far use one or moreprintfstatements directly. Aprintfcall is slow and reduces the throughput the BPFtrace program can monitor.This can be optimized by storing the data in map that is printed when BPFtrace is stopped. This will postpone the printf calls until observation is done. To do this, we introduce three new maps@start,@oid, and@vacuum. The first two maps are populated in the UProbe event of thevacuum_relfunction. The map@startcontains the time when the probe is triggered, and the map@oidcontains the Oid of the parameter function.When the function returns, and the URetProbe is triggered, the@vacuummap is populated. The key is the Oid, and the value is the needed time to perform the vacuum operation. Also, the keys of the first two maps are removed.When BPFtrace exits (i.e., by pressing CRTL+C), all populated maps are printed automatically. By using these three maps (@start,@oid,@vaccum), we have separated the actual monitoring from the output, the expensiveprintffunction is called after the monitoring is done.In addition, in the following program, we use the two functionsBEGINandENDthat are called by BPFtrace when the observation begins and ends.$ sudo sudo bpftrace -e '

uprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
{
        @start[tid] = nsecs;
        @oid[tid] = arg0;
}

uretprobe:/home/jan/postgresql-sandbox/bin/REL_14_2_DEBUG/bin/postgres:vacuum_rel
/@start[tid]/
{

        @vacuum[@oid[tid]] = nsecs - @start[tid];
        delete(@start[tid]);
        delete(@oid[tid]);

}

BEGIN
{
        printf(""VACUUM calles are traced, press CTRL+C to stop tracing\n"");
}

END 
{
        printf(""\n\nNeeded time in ns to perform VACUUM FULL per Oid\n"");
}
'After BPFtrace is started, the first message is printed. After the program is stopped, the second message is printed. Furthermore, the content of the@vacuummap is printed. For each Oid, the needed time for the vacuum operations is shown.VACUUM calls are traced, press CTRL+C to stop tracing
^C

Needed time in ns to perform VACUUM FULL per Oid

@vacuum[1153888]: 7526823
@vacuum[1153891]: 8462672
@vacuum[2613]: 10764797
@vacuum[2995]: 11429589
@vacuum[6102]: 11436539
@vacuum[12801]: 14373934
@vacuum[6106]: 14396012
@vacuum[3118]: 14507167
@vacuum[3596]: 14695385
@vacuum[12811]: 14871237
@vacuum[3429]: 15106778
@vacuum[3350]: 15158742
@vacuum[2611]: 15432053
@vacuum[3764]: 15534169
@vacuum[2601]: 16055863
@vacuum[3602]: 16128624
@vacuum[2605]: 16405419
@vacuum[2616]: 16914195
@vacuum[3576]: 17003920
[...]ConclusionThis article provides a brief overview of BPFtrace. To trace the function latency of PostgreSQL vacuum calls, we used the toolfunclatency-bpfcc.Additionally, we utilized BPFtrace to create a tool that allows for more in-depth observation of the calls. Our BPFtrace script also takes into account the parameters of the PostgreSQLvacuum_relfunction, enabling us to monitor the vacuum time per relation.And speaking of vacuum,check out this blog post to learn more about transaction ID wraparound exhaustion and how to avoid it in PostgreSQL databases.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/using-bpftrace-to-trace-postgresql-vacuum-operations/
2023-10-06T18:52:39.000Z,How to Reduce Your PostgreSQL Database Size,"Your phone buzzes in the middle of the night. You pick it up. A monitor went off at work—your PostgreSQL database is slowly but steadily reaching its maximum storage space. You are the engineer in charge. What should you do?Okay, if it comes down to that situation, you should probably remedy it ASAP by adding more storage. But you’re going to need a better strategy in the long run to optimize your PostgreSQL storage use, or you’ll keep paying more and more money.Does your PostgreSQL database really need to be that large? Is there something you can do to optimize your storage use?This article explores several strategies that will help you reduce your PostgreSQL database size considerably and sustainably.Why Is PostgreSQL Storage Optimization Important?Let’s spend a couple of minutes addressing this question first.Perhaps you’re thinking:“Storage is cheap these days, and optimizing a PostgreSQL database takes time and effort. I’ll just keep adding more storage.”Or perhaps:“My PostgreSQL provider is actually usage-based (like Timescale), and I don’t have the problem of being locked into a large disk.”Indeed, resigning yourself to simply using more storage is the most straightforward way to tackle an increasingly growing PostgreSQL database. Are you running servers on-prem? Slap another hard drive on that bad boy. Are you running PostgreSQL in RDS? Raise the storage limits. But this comes with problems.The first problem, and the most obvious, is the cost. If you’re running PostgreSQL in an EBS instance in AWS or inRDS, for example, you’ll be charged through an allocation-based model. This model assumes that you’ll need to predetermine how much disk space you’ll need in the future and then pay for it, regardless of whether you end up using it or not, and without the chance of downscaling.In other PostgreSQL providers, when you run out of storage space, you must upgrade and pay for the next available plan or storage tier, meaning that you’ll see a considerably higher bill overnight.In a way, these issues are mitigated by usage-based models.Timescalecharges by the amount of storage you use: you don't need to worry about allocating storage or managing storage plans, which really simplifies things—and the less storage you use, the less it costs.Usage-based models are a great incentive to actually optimize your PostgreSQL database size as much as possible since you’ll see immediate reductions in your bill. But yes, this also works the opposite way: if you don’t pay attention to managing your storage, your storage bill will increase.The second problem that comes with not optimizing your PostgreSQL storage usage is that this situation can lead to bad performance, with queries running slower and your I/O operations increasing. This is something that often gets overlooked,but maintaining PostgreSQL storage usage is paramount to keeping large PostgreSQL tables fast.‌‌This last point deserves a deeper dive into how data is actually stored in PostgreSQL and what is causing the problem, so let’s briefly cover some essential PostgreSQL storage concepts.Essential PostgreSQL Storage Concepts‌‌‌‌How does PostgreSQL store data?At a high level, there are two terms you need to understand: tuples and pages.A tuple is the physical representation of an entry in a table. You'll generally see the terms tuple and row used interchangeably. Each element in a tuple corresponds to a specific column in that table, containing the actual data value for that column.A page is the unit of storage in PostgreSQL, typically 8 kB in size, that holds one or more tuples. PostgreSQL reads and writes data in page units.Each page in PostgreSQL consists of a page header (which contains metadata about the page, such as page layout versions, page flags, and so on) and actual data (including tuples). There’s also a special area called the Line Pointer Array, which provides the offsets where each tuple begins.A simple representation of a PostgreSQL page containing metadata about the page and tuples stored in the pageWhat happens when querying data?When querying data, PostgreSQL utilizes the metadata to quickly navigate to the relevant page and tuple. The PostgreSQL query planner examines the metadata to decide the optimal path for retrieving data, for example, estimating the cost of different query paths based on the metadata information about the tables, indexes, and data distribution.What happens when we INSERT/ DELETE/ UPDATE a row in PostgreSQL?When a new tuple is inserted into a PostgreSQL table, it gets added to a page with enough free space to accommodate the tuple. Each tuple within a page is identified and accessed using the offset provided in the Line Pointer Array.If a tuple inserted is too big for the available space of a page, PostgreSQL doesn't split it between two 8kB pages. Instead, it employs TOAST to compress and/or break the large values into smaller pieces. These pieces are then stored in a separate TOAST table, while the original tuple retains a pointer to this external stored data.When we insert a tuple that's too large for a single page, a new page is created.What is a dead tuple?A key aspect to understand (and this will influence our PostgreSQL database size, as we’ll see shortly) is that when you delete data in PostgreSQL viaDELETE FROM,  you’re not actually deleting it but marking the rows as unavailable. These unavailable rows are usually referred to as “dead tuples.”When you runUPDATE, the row you’re updating will also be marked as a dead tuple. Then, PostgreSQL will insert a new tuple with the updated column.A page in a Postgres table with tuples that have been deleted or updated. The old instances are now dead tuplesYou might be wondering why PostgreSQL does this. Dead tuples are actually a compromise to reduce excessive locks on tables during concurrent operations, multiple connections, and simplifying transactions. Imagine a transaction failing halfway through its execution; it is much easier to revert a change when the old data is still available than trying to rewind each action in an idempotent way. Furthermore, this mechanism supports the easy and efficient implementation of rollbacks, ensuring data consistency and integrity during transactions.The trade-off, however, is the increased database size due to the accumulation of dead tuples, necessitating regular maintenance to reclaim space and maintain performance… What brings us to table bloat.What is table bloat?When a tuple is deleted or updated, its old instance is considered a dead tuple. The issue with dead tuples is that they’re effectively still a tuple on disk, taking up storage space—yes, that storage page that is costing you money every month.Table bloat refers to this excess space that dead tuples occupy in your PostgreSQL database, which not only leads to an inflated table size but also to increased I/O and slower queries. Since PostgreSQL runs under the MVCC system, it doesn't immediately purge these dead tuples from the disk. Instead, they linger until a vacuum process reclaims their space.Table bloat also occurs when a table contains unused pages, which can accumulate as a result of operations such as mass deletes.A visualization of table bloat in PostgreSQL. Pages contain many dead tuples and a lot of empty spaceWhat isVACUUM?Dead tuples get cleaned and deleted from storage when theVACUUMcommand runs:VACUUM customers;Vacuumhas a lot of roles, but the relevant point for this article is that vacuum removes dead tuples once all connections using the dead tuples are closed.VACUUMby itself will not delete pages, though. Any pages created by a table will stay allocated, although the memory in those pages is now usable space after running vacuum.What is autovacuum?Postgres conveniently includes a daemon to automatically run vacuum on tables that get heavy insert, update, and delete traffic. It operates in the background, monitoring the database to identify tables with accumulating dead tuples and then initiating the vacuum process autonomously.Autovacuum comes enabled by default, although the threshold PostgreSQL uses to enable autovacuum is very conservative.What is VACUUM FULL?Autovacuum helps with dead tuples, but what about unused pages?TheVACUUM FULLcommand is a more aggressive version ofVACUUMthat locks the table, removes dead tuples and empty pages, and then returns the reclaimed space to the operating system.VACUUM FULLcan be resource-intensive and requires an exclusive lock on the table during the process. We’ll come back to this later.Now that you have the necessary context, let’s jump into the advice.How To Reduce Your PostgreSQL Database SizeUse Timescale compressionThere are different ways we can compress our data to consistently save storage space.PostgreSQL has some compression mechanisms, but if you want to take data compression even further, especially for time-series data, you should useTimescale’s columnar compression.It allows you to dramatically compress data through a providedadd_compression_policy()function. To achieve high compression rates,Timescale uses various compression techniquesdepending on data types to reduce your data footprint. Timescale also uses column stores to merge many rows into a single row, saving space.Let's illustrate how this works with an example.Let’s say we have ahypertablewith a week's worth of data. Imagine that our application generally only needs data from the last day, but we must keep historical data around for reporting purposes. We could runSELECT add_compression_policy('my_table', INTERVAL '24 hours');which automatically compresses rows in themy_tablehypertable older than 24 hours.Timescale’s compression would combine all the rows into a single row, where each column contains an array of all the row's data in segments of 1,000 rows. Visually, this would take a table that looks like this:| time                   | location | temperature |
|------------------------|----------|-------------|
| 2023-09-20 00:16:00.00 | garage   | 80          |
| 2023-09-21 00:10:00.00 | attic    | 92.3        |
| 2023-09-22 00:5:00.00  | basement | 73.9        |And compress it down to a table like this:| time                                                                     | location                    | temperature               |
|--------------------------------------------------------------------------|-----------------------------|---------------------------|
| [2023-09-20 00:16:00.00, 2023-09-21 00:10:00.00, 2023-09-22 00:5:00.00]  | [garage, attic, basement]   | [80, 92.3, 73.9]          |To see exactly how much space we can save, let's run compression on a table with 400 rows, 50 rows per day for the last seven days, that looks like this:CREATE TABLE conditions (
  time        TIMESTAMPTZ       NOT NULL,
  location    TEXT              NOT NULL,
  temperature DOUBLE PRECISION  NULL,
);

SELECT create_hypertable('conditions', 'time');Next, we'd add a compression policy to run compression onconditionsfor rows older than one day:SELECT add_compression_policy('conditions', INTERVAL '1 day')In the Timescale platform, if we navigate to theExplorer tabunder Services, we’d see our table shrink from 72 kB to 16 kB—78% savings!The Timescale console showing a 78% space reduction in table size due to compressionThis is a simple example, but it shows the potential that Timescale compression has to reduce storage space.Monitor dead tuplesA great practice to ensure you’re using as little storage as possible is to consistently monitor the number of dead tuples in each table.This is the first step towards putting together an efficient PostgreSQL storage management strategy.To see pages and tuples in action, you can usepgstattuple(), an extension provided by the Postgres maintainers to gain insights into how our tables manage tuples:CREATE EXTENSION IF NOT EXISTS pgstattuple;If you run the following query,SELECT * FROM pgstattuple('my_table');ccPostgres would give you a table of helpful information in response:table_len | tuple_count | tuple_len | tuple_percent | dead_tuple_count | dead_tuple_len | dead_tuple_percent | free_space | free_percent 
-----------+-------------+-----------+---------------+-----------------+----------------+--------------------+------------+--------------
  81920000 |      500000 |  40000000 |          48.8 |           10000 |        1000000 |                1.2 |     300000 |          0.4table_lentells you how big your table is in bytes, including data, indexes, toast tables, and free space.dead_tuple_lentells how much space is being occupied by dead tuples which can be reclaimed by vacuuming.free_spaceindicates the unused space within the allocated pages of the table.. Take note thatfree_spacewill reset for every new page created.You can also perform calculations or transformations on the result to make the information more understandable. For example, this query calculates the ratios of dead tuples and free space to the total table length, giving you a clearer perspective on the storage efficiency of your table:SELECT
(dead_tuple_len * 100.0 / table_len) as dead_tuple_ratio,
(free_space * 100.0 / table_len) as free_space_ratio
FROM
pgstattuple('my_table');Run autovacuum more frequentlyIf your table is experiencing table bloat, having autovacuum run more frequently may help you free up wasted storage space.The default thresholds and values for autovacuum are inpostgresql.conf. Updatingpostgresql.confwill change the autovacuum behavior for the whole Postgres instance. However, this practice is generally not recommended, since some tables will have a higher affinity for dead tuples than others.Instead, you should update autovacuum's settings per table. For example, consider the following query:ALTER TABLE my_table SET (autovacuum_vacuum_scale_factor = 0, autovacuum_vacuum_threshold = 200)This will updatemy_tableto have autovacuum run after 200 tuples have been updated or deleted.More information about additional autovacuum settings are in thePostgreSQL documentation. Each database and table will require different settings for how often autovacuum should run, but running vacuum often is a great way to reduce storage space.Also, keep an eye on long-running transactions that might block autovacuum, leading to issues. You can use PostgreSQL’spg_stat_activityview to identify such transactions, canceling them if necessary to allow autovacuum to complete its operations efficiently:SELECT pid, NOW() - xact_start AS duration, query, state
FROM pg_stat_activity
WHERE (NOW() - xact_start) > INTERVAL '5 minutes';

#Cancelling
SELECT pg_cancel_backend(pid);You could also inspect long-running vacuum processes and adjust theautovacuum_work_memparameter to increase the memory allocation for each autovacuum invocation,as we discussed in our article about PostgreSQL fine tuning.Reclaim unused pagesAutovacuum and vacuum will free up dead tuples, but you’ll need the big guns to clean up unused pages.As we saw previously, runningVACUUM FULL my_tablewill reclaim pages, but it has a significant problem: it exclusively locks the entire table. A table runningVACUUM FULLcannot be read or written to while the vacuum has the lock, which can take a long time to finish. This is usually an instant no-go for any production database.The PostgreSQL community has a solution,pg_repack.pg_repackis an extension that will clean up unused pages and bloat from a table by cloning a given table, swapping the original table with the new table, and then deleting the old table. All these operations are done with minimal exclusive locks, leading to less downtime.At the end of thepg_repackprocess, the pages associated with the original table become deleted from storage, and the new table only has the absolute minimum number of pages to store its rows, thus freeing table bloat.Find unused indexesAs we mentionin this article on idexing design, over-indexing is a frequent issue in many large PostgreSQL databases. Indexes consume disk space, so removing unused or underutilized indexes will help you keep your PostgreSQL database lean.You can usepg_stat_user_indexesto spot opportunities:SELECT
relname AS table_name,
indexrelname AS index_name,
pg_size_pretty(pg_relation_size(indexrelid)) AS index_size,
idx_scan AS index_scan_count
FROM
pg_stat_user_indexes
WHERE
idx_scan < 50 -- Choose a threshold that makes sense for your application.
ORDER BY
index_scan_count ASC,
pg_relation_size(indexrelid) DESC;(This query looks for indexes with fewer than 50 scans, but this is an arbitrary number. You should adjust it based on your own usage patterns.)Arrange columns by data type (from largest to smallest)In PostgreSQL, storage efficiency is significantly influenced by the ordering of columns, which is closely related to alignment padding determined by the size of the column types. Each data type is aligned at memory addresses that are multiples of their size.This alignment is systematic, ensuring that data retrieval is efficient and that the architecture adheres to specific memory and storage management protocols. But this can also lead to unused spaces, as the alignment necessitates padding to meet the address multiple criteria.The way to fix this is to strategically order you columns from the largest to the smallest data type in your table definitions. This practical tip will help you minimize wasted space.Check out this article for a more in-depth explanation.Delete old data regularlyYou should always ask yourself: how long should I keep data around? Setting up data retention policies is essential for managing storage appropriately. Your users may not need data older than a year ago. Deleting old, unused records and indexes regularly is an easy win to reduce your database size.Timescale can automatically delete old data for us usingretention policies. Timescale’s hypertables areautomatically partitioned by time, which helps a lot with data retention.  Retention policies automatically delete partitions (which are called chunks in Timescale) once the data contained in such partition is older than a given interval.You cancreate a retention policyby running:SELECT add_retention_policy('my_table', INTERVAL '24 hours');In this snippet, Timescale would delete chunks older than 24 hours frommy_table.Wrap-UpWe examined how table bloat and dead tuples can contribute to wasted storage space, which not only affects your pocket but also the performance of your large PostgreSQL tables.To make sure you’re reducing your PostgreSQL database size down to its minimum, make sure to enable Timescale compression, to use data retention policies, and to set up a maintenance routine to periodically and effectively delete your dead tuples and reclaim your unused pages.All these techniques together provide a holistic approach to maintaining a healthy PostgreSQL database and keeping your PostgreSQL database costs low.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-reduce-your-postgresql-database-size/
2023-01-18T16:23:18.000Z,What's New in TimescaleDB 2.9?,"The latest version of TimescaleDB 2.9 is now available onTimescaleand as an upgrade option foron-premise installations. This major release includes the following new features:Hierarchical continuous aggregates:you can now create and use a continuous aggregate on top of another continuous aggregate.Background jobs:introducing fixed schedules for background jobs and the ability to check and troubleshoot job errors.Hyperfunction improvements:improvedtime_bucket_gapfillfunction allows to specify the time zone to a bucket.Multi-node improvements:use ofalter_data_node()to change the data node configuration. With this function, you can now configure the availability of the data node.Let’s explore these improvements in more detail.Welcome, Hierarchical Continuous AggregatesWith TimescaleDB 2.9 and its later versions, users can downsample their time-series data more efficiently and faster than before by creating continuous aggregate hierarchies.Eachhierarchical continuous aggregateis defined on top of the previous one, providing a lower granularity view of the same dataset while making the refresh process for higher-level continuous aggregates lightning-fast. This also enables TimescaleDB users to define more diverse sets of continuous aggregates without significant additional costs.For example, a continuous aggregate providing a daily summary only needs to go through 24 records each time it is refreshed when built on top of a continuous aggregate that provides an hourly summary—forget those tens of thousands of records stored in the raw hypertable for that day.Imagine that you have a finance dashboard, as shown in the example below. With hierarchical continuous aggregates, you can visualize your stock status by week by using pre-aggregated daily data (without going through raw data in the hypertable). This results in super fast response times, which prevent a laggy user experience of the dashboard.Diagram example of hierarchical continuous aggregates functionality for a finance use caseFor a more in-depth read about hierarchical continuous aggregates, their examples, and code snippets, read Chris Englebert’s blog post “An Incremental Materialized View on Steroids: How We Made Continuous Aggregates Even Better.”More Scheduling Options and Better Visibility for Policies and User-defined ActionsTimescaleDB natively includes support for automation policies, such as the following:Continuous aggregate policiesto automatically refresh continuous aggregatesCompression policiesto compress historical dataRetention policiesto drop historical dataReordering policiesto reorder data within chunksThose policies, together with user-defined actions, are meant to run at regular intervals. At a high level, they are jobs scheduled to run asynchronously in the background, facilitated by our job scheduler.With TimescaleDB 2.9, you can now set jobs to run at specific times by using our newly introduced fixed scheduling semantics. For example,by setting a continuous aggregate policyto start at03:00with a 24-hour schedule interval, the job will always be executed at 3:00 a.m. every day.We also introduced better visibility by providing information aboutruntime errors encountered by jobsrun by the automation framework.Other Improvements in TimescaleDBWe also continued to improve thetime_bucket_gapfillfunction to allow specifying the time zone to a bucket and introduced multi-node improvements. To be more precise, we improved the use of thealter_data_node()function by introducing the option to configure the availability of the data node.Try TimescaleDB 2.9If you are using Timescale, upgrades are automatic, and you’ll be upgraded to TimescaleDB 2.9.1 in your next maintenance window.New to Timescale?Start a free 30-day trial, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/whats-new-in-timescaledb-2-9/
2022-12-06T14:00:48.000Z,How to Design a Better Developer Experience for Time-Series Data: Our Journey With Timescale's UI,"Part of the Timescale mission is to create a seamless developer experience for those working with time-series data—something that user experience and user interface (UX/UI) design play a massive role in. With Timescale, we're building a cloud-native database platform from scratch. Behind it, there’s a multi-disciplinary team of designers, product managers, and engineers working closely together to build, iterate, and progressively improve our platform. (By the way, hi 👋, this is the Product Design Team speaking!)Designing the user interface of a modern database platform is a peculiar challenge. As with any great design, you need to prioritize the user (or, in this case, the developer) experience, optimizing for the wide and diverse range of workflows within the platform. Our customers build many things with Timescale, from powering their frontend dashboards to running complex analytics.Some database platforms are very complicated to manage, which we wish to avoid by making things as easy and seamless as possible for the user. Simultaneously, we aim to keep the control in the hands of the developer—we strive for transparency and openness, steering away from black boxes.Today, we’re sharing our perspective on how to solve these problems by taking you on a journey through the evolution of the Timescale Cloud UI. We’ll start right at the beginning, when our product was called Timescale Forge (remember those days?), all the way up to its current state, closing it off with a few lessons we learned along the way.P.S.When we wrote this blog post, the UI’s state was very recent 👀🔥 as we just launched an important UX redesign that took us many months to complete, which we dubbed Timescale 2.0.P.S.S.Since our Timescale 2.0 redesign, we have completely rebranded, including, of course, our UI. Our product name has also changed from Timescale Cloud to a cleaner Timescale, which is also our brand name. Check outour websitefor the changes, andread more about the rebranding in this blog post—we promise to share our design lessons from the rebranding sometime!But now, let us tell you the whole story (up to Timescale 2.0). It has been a fun rollercoaster teamwork effort, and we want to share it with you!Improving Developer Experience: Devising 2.0If you are not a Timescale user or don’t know us, let us give you a little context first (if you already do,feel free to jump to the next section).Timescaleis a modern, cloud-native relational database platform for time series and analytics built onPostgreSQL.If you’re not a developer and this sounded like advanced math, let’s take the engineering jargon down a notch: we are a cloud product that allows developers to ingest, store, and analyze time-series data. (Time-series data is a collection of repeated measurementsthat will enable you to track changes over time. You will find it in health apps that check your pulse, in the finance sector to monitor stock markets, or in smart home devices, for example.)Timescale is fast (with 350x faster queries and 44 % faster ingest than the competition) and simple to get started (you can spin up an instance in 30 seconds). Also, it offers advanced functionality to help store high volumes of time series data for less money (likecolumnar compressionordata tiering to S3). The product is built on top of PostgreSQL, one of the most reliable and loved open-source relational databases, and it runs in AWS.From a simplistic point of view, progressively improving the design of a database product like Timescale may look like a simple facelift. But, as we mentioned, developer experience is vital to us, so the entire design evolution process comprised more than visual changes to make it “prettier.” Our goal was to improve the user experience dramatically.So, we spent months planning, researching, understanding user paths and pain points, iterating, debating, iterating, testing, iterating, and drinking many cups of coffee (and tea!) before launching the massive Timescale 2.0 redesign.To correctly tackle a redesign, you can’t redesign for yourself according to your assumptions and sense of style. We did an in-depth analysis, and for every decision, we considered the user needs and our team’s needs, the user pain points, the user experience, the company goals, the brand, our value proposition, etc. We needed to be entirely sure about the decision to redesign our platform because it is a long and challenging—but also fun!—journey once you start!But, we didn't redesign the layouts just for a better developer experience. We hoped to build a solid foundation on both the design and development sides. We improved engineering velocity by switching all the design elements to reusable components. This means that some parts we created from scratch and others we re-examined completely: UI library, user flows, a storybook with all the components, and a new code.But why did we make this decision? What was wrong with the interface? How was the process? How long did it take to make the changes?Let’s go back in time to answer all these questions.Episode I: The Timescale Forge EraWhen we created Timescale, it had a different name: Timescale Forge. The product was started in 2019 when we had less than 30 people in the company, and of course, the UI Team was incredibly small. In the beginning, the Design Team comprised only one person, and the Frontend Team was only three (working on the product and everything related to Timescale), so all the work was admirable.Timescale Forge was the first minimum viable product (MVP) of this fantastic product we are still building, which we call Timescale Cloud (at the time of writing this blog post). But before swapping names, in early 2021, the company started to grow, so we created a plan to improve the interface, functions, user experience, and UI structure. We analyzed the interface and detected some problems:We were using a small piece of space on the screen to interact.We had tiny font sizes to fit that small space.The functions and user experiences needed to be more consistent.We made less-than-ideal choices in some of the interactions.The UI was stiff and old-school.This is how our interface looked then:0:00/1×Timescale Forge, March 2021Creating a service in the old Timescale ForgeAnd these are some of the problems we identified:A. There was a lot of dead space.B. We had an important (call to action) CTA with a small font size (10 pt).C. Some of the instructions looked like a banner.D. We had some fonts in size 10 pt.E. There were some nonsense actions (like clicking “Next” after completing your name).F. We used a lot of monospace fonts.G. We confused users by applying an orange-gold color (usually associated with warnings) to interactions.H. The warning color (brown) was very close to the main color.Our Design and Frontend teams were smaller then, so we knew we couldn’t handle all the external and internal improvements we wanted. So, we divided them into baby steps.The first step we made was to increase the screen space and make the font sizes bigger to optimize our users' experience and legibility.Being legible means ""that can be read."" Legibility is determined by respecting the visual design and typography rules (we didn’t have much then), such as good contrast, tracking, kerning, and leading.Some rules to have excellent legibility include the following:Use a reasonably large default font size: we used small font sizes, and the screen was hard to read. We had some text in 10 pt or even 8 pt.Create high contrast between characters and background: we changed some grays to darker grays to avoid low contrast.Use a clean typeface: our principal font is Inter, a clear and simple font. But we also use JetBrain (a monotype font). One of our changes to increase legibility was to define clear use cases for the monotype font instead of using it randomly because it can sometimes be hard to read.Based on these rules, we implemented a few changes to the Timescale Forge UI, which you can see in this video:0:00/1×A comparison of Timescale Forge between March 2021 and June 2021As you can see, we only fixed a few elements:A. Less dead spaceB. A larger CTA (14 pt) that’s disabled when it needs to be disabledC. No changesD. Increased font size from 10 pt to 14 ptE. Removal of nonsense actions (like clicking “Next” after completing the name)F. No changesG. No changesH. No changesWhile this small Forge redesign may not seem very powerful, low legibility made our product hard to use, and a product that is hard to use is not a good product. However, we were determined to make it even better, particularly as a name change loomed on the horizon.Episode II: From Timescale Forge to Timescale CloudAfter this minor update in July 2021, we changed the name of our platform from Timescale Forge to Timescale Cloud— of course, that meant we needed to revisit the UI.With the product renaming, we had to change the logo, so it felt like a fantastic opportunity to update the design. But we didn’t have the time to do a full-blown overhaul in just a couple of months before the announcement of the new platform.Still, the seed of a significant redesign sprouted at this moment. We knew the change should be fast, but we wanted to clear the path for the new interface. So, we did a UI exercise! Each designer scanned the interface for two to three things they wanted to change and then explored the different visual paths. The idea was to set a vision for the future through fun exploration so that we could find a base for the next iteration.Following this exploratory exercise, the team combined all their options and built something similar to the layout that we wanted to implement in the future. Then, we took a step back to make a streamlined update where we immediately applied these changes.The results? A new logo, colors, and minor changes made the interface softer and more inviting.The exercise also opened the door to discussions about the look and feel of the redesign, project timelines, and how it would affect the overall product.But before moving on to the actual redesign, check out our interface evolution! This is the before:Timescale Forge overview between March 2021 and June 2021The in-between phase, while we were working on our UI exercise:Timescale Cloud overview—UI exercise to start envisioning the futureAnd this is the after version:Timescale Cloud overview, September 20210:00/1×Episode III: The Rise of 2.0In September 2021, afterthe launch of the new Timescale Cloud, we found the perfect timing to officially start planning the next iteration, the so-called Timescale 2.0. It was all about our end goal: to create a better interface and developer experience for our users and a better structure for us. So we sat down—the Product Design, Product, and Engineering teams—and started planning everything.Let’s share what each of us had in mind.Design: It's all about the developer experienceOn the design side, we decided to create the following:1. A newdesign systemand UI libraryin Figma to have a guide with all the rules, specifications, and reusable components to build our interface.2. Redesign the interface from scratch.We kept some functionalities and changed others, but we redesigned the layout completely.3. Become fully user-centered.Instead of designing for the entire user journey, we had only focused on single screens because we started with a tiny team without the capacity to work full-time in the Product area. Now, we always deploy the flows taking into account the complete user path.4.Redesign the illustrations.Our company's mascot (Eon) is a tiger, and we decided to create a flat version and integrate it better into our product.Engineering: The right tools for your libraryAnd this is what we did on the development side:1. We built a frontend librarythat mapped to Figma as closely as possible. Each component either directly matched a design component or, where the underlying logic was the same, matched several components using styling variants.Having reusable components is a common development pattern these days, but implementing them well requires a solid maintenance plan. Developers need to be able to, at a minimum: know what components are available and how components function, make changes quickly and check whether changes to one component have unforeseen effects elsewhere.Creating mostly reusable components also forced us to be more diligent about separating display logic from app logic, allowing us to easily swap in new styles if we need to make changes.2. We moved toStorybook, a custom-made tool for organizing components. The idea was to have the same well-organized library for both teams but in different applications (Figma vs. Storybook). Using both, we can maintain the consistency of the product and save time creating and reviewing new flows.3.We adoptedChromatic,the cloud-based version of Storybook. The main benefit this brought us was another way to connect design and development. Developers can run Storybook locally while they build, but any proposed changes to a component in the library on the developer-side surfaces on the Chromatic site.Chromatic auto-builds the library for each change and highlights the differences between that component and any dependent components. This makes it easy to view and discuss changes between the two teams.We could have built something ourselves to host the Storybook library, but there’s always a trade-off between paying someone else to do a task and paying for development and maintenance time to do it yourself. In this case, the folks at Chromatic are the same folks that put out Storybook and are constantly improving their product, so we felt comfortable letting them do what they do best!4.We converted most of our code fromJavaScripttoTypeScript. This doesn’t directly affect the component design, but it gave us several benefits regarding work processes. TypeScript gives JavaScript static type checking, which generally reduces errors while passing variables between components (and functions in general). It also makes it easy to specify the format of arguments a function should receive. All of this allows us to build new things with more confidence.Our guiding principlesWe couldn’t have completed these tasks without a few guidelines, and two emerged as especially crucial for this project. The first was building a treasure trove of design and development elements to speed up the creation process—our libraries— and the second was embracing a fully user-centered design.Here’s why these matter to us.Why is it important to have a library?When you need to move fast, improve, and iterate constantly (as is common in startups), it is normal to have tons of inconsistencies throughout the product. It is hard to keep the correct balance between shipping fast and making it all perfect and consistent without any technical debt. That’s what happened to us.But if you have a robust design system and library, you can minimize and avoid inconsistencies. Another significant benefit is reducing code duplication. If a component exists and is easy to find, it prevents you from rebuilding that component somewhere else in the product. So, those files will become your source of truth with all of the product requirements to iterate and build new flows. You can adequately set up the text styling, colors, paddings, shadows, containers, distances, icons, illustrations, behaviors, motion, breakpoints, etc.Besides setting up all the rules, a library makes design and development work much faster because you already have the components built and can reuse them. So, you may ask, if we needed to move fast, why didn’t we have a library from the beginning? And, yes, that question makes sense. We all should have libraries to work better and faster from the start. But the truth is that creating a library takes a lot of time, so that’s why it took us so much to finally have a good library.But, basically, with a library, you have all the elements already built. If someone needs to update something, you need to edit the mother file (on both the design and development sides), and that change will impact all of your product screens, keeping up the consistency on top of everything. And consistency in a product is another key to a good product.Preview of the Timescale UI Library—FigmaWould you like to have a sneak peek at our UI Library? Click here to see aninteractive prototype.Preview of the Timescale StorybookAnother game-changer: Embrace user-centered designWhen we design a digital product, it is vital to understand who the user is and how they will use it. If we don’t include the user in the picture, it’s like you are watching only half of the movie.Having a user-centered design means that our processes put the user always as the first thing to discuss. We take into account user requirements, objectives, and desires. Satisfying users’ needs becomes a priority, and all the decisions are evaluated in the context of whether it delivers value to the users or not.First, we need to change how we think and, after that, change how we work. For example, we changed something as small as how we deploy the flows on Figma to embrace this new methodology. We used to design by screens/sections, and now we deploy the complete user flow (starting with the happy path) to understand the full picture, catch all the user corner cases and avoid dead-end roads with no solutions for the users. We are not masters of user-centered design yet, but we are starting to incorporate it into our daily workflow to improve our product.Episode IV: Ready to LaunchSince we began working on 2.0, we have gone through many iterations, prototypes, stakeholder meetings, updates, plans, estimates, designs, etc. More than one year may sound like a crazy timeline to work on a redesign, but we did it while improving our current interface. You can’t stop updating the product, releasing new features, and incorporating new elements just because you are working on a redesign.What you need is to change the car’s wheels while in motion. So, for more than one year, the team worked simultaneously on two paths: moving forward with Timescale 1.0 (that’s what we call it internally) and building the new features of Timescale 2.0.Several people asked us why we didn’t break such a massive project into smaller pieces to release it faster. We split the new proposal into MVPs and future iterations, but we wanted to make a big first release for two reasons. The first is technical incompatibilities. The development team built the new interface with an improved structure and code, making it impossible to release it all at once (so stay tuned because further improvements are coming soon!).And the other reason is focused on the developer experience: the new interface looks and feels different, and having both simultaneously would give users the idea that we’re not consistent in our product. Plus, some screens could look broken.But, after a year’s work, we finally launched 2.0, and we are ready to show it to you. 🥁0:00/1×This is how the Services page looks now in Timescale Cloud. This is the first screen you’ll see when you log in, where you can switch between your different Timescale databases.Timescale, Services page, December 2022When you click on your Services, you’ll land on the Overview page, which now lets you see a glimpse of the key info for your database configuration, service metrics, and pricing. We also made this screen actionable: if you want to resize your service, for example, you can click on the contextual menu (those three little dots) on the left of the storage card, for example, and simply add more storage to your service.Timescale Cloud overview, December 2022 (Did anyone catch The Office reference?)Another page we’d like to show you is the Explorer screen. This compiles information about your database internals—tables and hypertables, compression, continuous aggregates, etc.And in the Operations tab, you canseamlessly resize your database;configure storage autoscaling;add replicas for high availability and/or for scaling reads;configure VPC peering;upgrade your major Postgres version;request access to data tiering, and much more.So, first thoughts? We know we’re biased, but we’re very happy with what we achieved and hope our users will enjoy it as much as we do!Want to have a closer look at our interface?Browse our interactive prototype! Or, if you want to go beyond looks and get a taste of the frictionless developer experience we created for you while reaping all the benefits of a modern, cloud-native relational database platform for time series,you can try Timescale for free for 30 days (no credit card required).Epilogue: The Journey Has Just BegunAs with so many journeys, ours wasn’t over when we implemented the new redesign and structure—it had only begun. Now that we had everything in place, we started the last-mile work of creating a launch strategy, updating the documentation, testing the interface with internal users, training our internal teams, preparing material for the launch (like this blog post), etc.This is the first lesson we can share about such massive projects: our evolutionary journey has just started. We have to continue iterating, educating users and our colleagues, and promoting our new interface.But after more than a year’s work where we experienced all possible emotions and learned so much, there are a few more good lessons:Never underestimate work: realistic planning is key.Since starting this initiative, we have probably made a million estimates. Rebuilding everything takes time, especially when you are building a better structure and constantly improving the flows because you keep shipping new features. So, never underestimate your work!The dream team: product managers, engineers, and designers working together.It took us a couple of months to reach the desired level of teamwork for this project because it is hard to prioritize a redesign when you have new features to launch. But here’s the deal: great planning and teamwork are key to moving forward.You don’t need to complete your designs before starting to code. The Design Team waited until we had the UI library and all the flows’ drafts ready to make the official hand-off to the Engineering Team. Looking back, we probably could have saved some time and completed the hand-off while still working on the flows. But, it seemed right to do it that way because we were launching other features, and we didn’t want to “steal” development efforts when our flows weren’t yet closed.Break the process into manageable tasks.We did this from the beginning, which was the best way to go. We divided everything into manageable items on GitHub: flows, components, iterations, quality assurance, etc., and we split the assignments internally between the team. A concrete task is not as overwhelming and has a better chance of completion in less time.Learn when to say “yes” to feedback and when to say “let’s come back to it after the launch.”Receiving feedback is vital to any project because, with every input, the design grows stronger and more logical. More eyes are better than just a few. But you need to know when to stop receiving feedback, or you’ll never end the project. So learn when to ask for feedback and when to save issues for future iterations, focusing only on MVPs.And, speaking of feedback, we would be happy to hear your thoughts about Timescale 2.0! You can reach out on ourCommunity Slackand leave feedback so we can continue to improve our developer experience for all our users. And watch this space: as we said, our journey has just begun, and we have more UI news coming up soon!Haven’t tried Timescale yet and would like to get a look (and feel) of our new UI?Join us in a 30-day free trialfor a trip down faster queries, worry-free operations, and of course, a seamless developer experience.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-to-design-a-better-developer-experience-for-time-series-data-our-journey-with-timescales-cloud-ui/
2023-01-19T16:28:52.000Z,The PostgreSQL Job Scheduler You Always Wanted (Use it With Caution),"As a PostgreSQL guy, it really makes you wonder why a built-in job scheduler is not a part of the core PostgreSQL project. It is one of the most requested features in the history of ever. Yet, somehow, it just isn’t there.Essentially, a job scheduler is a process that kicks off in-database functions and procedures at specified times and runs them independently of user sessions. The benefits of having a scheduler built into the database are obvious: no dependencies, no inherent security leaks, fits in your existing high availability plan, and takes part in your data recovery plan, too.ThePostgreSQL Global Development Grouphas been debating for years about including a built-in job scheduler. Even after the addition of background processes that would support the feature (all the way back in 9.6), background job scheduling is unfortunately not a part of core PostgreSQL.So being the PostgreSQL lovers we are atTimescale,we decided to build such a schedulerso that our users and customers can benefit from a job scheduler in PostgreSQL. In TimescaleDB 2.9.1, we extended it to allow you to schedule jobs with flexible intervals andprovide you with better visibility of error logs.The flexible intervals enable you to determine whether the next run of the job occurs based on the scheduled clock time or the end of the last job run. And by “better visibility” of the job logs, we mean that they are also being logged to a table where they can be queried internally. These were extended to prevent overlapping job executions, provide predictable job timing, and provide better forensics.We extensively use the advantage of this internal scheduler for our core features, enabling us to defercompression,data retention, and refreshing of continuous aggregates to a background process (among other things).📝 Editor's note:Learn more about how TimescaleDB's hypertables enable all these features above as a PostgreSQL extension, plus other awesome things like automatic partitioning.This scheduler makes Timescale much more responsive to the caller and results in more efficient processing of these tasks. For our own benefit, the job scheduler needs to be internal to the database. It also needs to be efficient, controllable, and scale with the installation.We made all this power available to you as a PostgreSQL end user. If you're running PostgreSQL in your own hardware, you caninstall the TimescaleDB extension. If you're running in AWS,you can try our platform for free.The PostgreSQL Job Scheduler DebateBut not so fast. Before you start rejoicing, let’s review the reasons that the PostgreSQL Global Development Group chose not to include a scheduler in the database—there'll be educational for you as a word of caution.Rather than rehashing the discussion list on the subject, let's summarize the obstacles that came up in themailing list:PostgreSQL is multi-process, not multi-thread.This simple fact makes having a one-to-one relationship of processes to user-defined tasks a fairly heavy implementation issue. Under normal circumstances, PostgreSQL expects to lay a process onto a CPU (affinity), load the memory through the closest non-uniform memory access (NUMA) controller, and do some fairly heavy data processing.This works great when the expectation is that the process will be very busy the majority of the time. Schedulers do not work like that. They sit around with some cheap threads waiting to do something for the majority of the life of the thread. Just the context switching alone would make using a full-blown process very expensive.Background workers' processes are a relatively small pool by design.This has a lot to do with the previous paragraph, but also that each process allocates the prescribed memory at startup. So, these processes compete with SQL query workers for CPU and memory. And the background processes have priority over both resources since they are allocated at system startup.The next issue is more semantic.There are quite a few external schedulers available. Each one of them has a different implementation of the time management system. That is, there is a question about just how exactly the job should be invoked. Should it be invoked again if it is still running from the last time? Should the job be started again based on clock time or relative to the previous job run? From the beginning or the end of the last run?There are quite a few more questions of this nature, but you get the idea. No matter how the community answers these questions, somebody will complain that the implementation is the wrong answer because\<insert silly mathematician answer here\>.Why We Still Need a PostgreSQL Job SchedulerTimescale doesn't have the luxury of debating how many angels can dance on the head of a pin. As a database service working with large volumes of data in PostgreSQL, we face a hard requirement of background maintenance for the actions of archival, compression, and general storage. Timescale's core features, excludinghyperfunctions, depend on the job scheduler.But, rather than create a bespoke scheduler for our own purposes we built a general-purpose scheduler with a public application programming interface.This general-purpose scheduler is generally available as part of TimescaleDB. You may use it to set a schedule for anything you can express as a procedure or function. In PostgreSQL, that's a huge advantage because you have the full power of the PostgreSQL extension system at your disposal. This list includes plug-in languages, which allow you to do anything the operating system can do.Timescale assumes that the developer/administrator is a sane and reasonable person who can deal with a balance of complexity. That is longhand for ""we trust you to do the right thing.""With Great Power Comes Great ResponsibilitySo, let's talk first about a few best design practices for using the Timescale (PostgreSQL) built-in job scheduler.Keep it short.The dwell time of the background process can lead to high concurrency.  You are also using a process shared by other system tasks such as sorting, sequential scans, and other system tasks.Keep it unlocked.Try to minimize the number of exclusive locks you create while doing your process.Keep it down.The processes that you are using are shared by the system, and you are competing for resources with SQL query worker processes. Keep that in mind before you kick off hundreds or thousands of scheduled jobs.Now, assuming we are using the product fairly and judiciously, we can move on to the features and benefits of having an internal scheduler.Built-In PostgreSQL Job Scheduler: All the Nice StuffNow that we've covered the things that demand caution, here's a list of some of the benefits of using this scheduler:Physical streaming replication will also replicate the job schedule. When you go to switch over to your replica, everything will already be there.You don't need a separate high-availability plan for your scheduler. If the system is alive, so are your scheduled jobs.The jobs can report on their own success or failure to internal tables and the PostgreSQL log file.The jobs can do administrative functions like dropping tables and changing table structure by monitoring the existing needs and structures.When you install Timescale, it's already there.📝Editor's note: Quick reminder that you caninstall the TimescaleDB extensionif you're running your own PostgreSQL database, orsign up for the Timescale platform(free for 30 days).How The Job Scheduler WorksThere isa quick introductory article in the Timescale documentation. Click that link if you want more detailed information.The TL;DR version is that you make a PostgreSQL function or procedure and then call theadd_job()function to schedule it. Of course, you can remove it from the schedule using… Wait for it...delete_job().That's it. Really. All that power is at your fingertips, and all you need to know is two function signatures.Something to be aware of while you're using the scheduler is that the job may be scheduled to repeat from the end of the last run or from the scheduled clock time (in TimescaleDB 2.9.1 and beyond). This allows you to ensure that the previous job has completed (by picking from the end of the run) or that the job executes at a prescribed time (making job completion your responsibility).If you feel a bit homesick and just want to look at your adorable job, there's also:SELECT * FROM timescaledb_information.jobs;And, of course, for completeness, there's alwaysalter_job()for rescheduling, renaming, etc.Once your job has been created, it becomes the responsibility of the job scheduler to invoke it at the proper time. The job scheduler is a PostgreSQL background process. It wakes up every 10 seconds and checks to see if any job is scheduled in the near future.If such a job is queued up, it will request another background process from the PostgreSQL master process. The database system will provide one (provided there are any available). The provided process becomes responsible for the execution of your job.This basic operation has some ramifications. We have already mentioned that we need to use these background processes sparingly for resource allocation reasons. Also, there are only a few of them available. The maximum parallel count of background processes is determined bymax_worker_processes. If you need help configuring TimescaleDB background workers,check out our documentation.📝 You can also check out this blog post on tuning TimescaleDB parameters.On my system (Kubuntu 22.04.1, PostgreSQL 14.6), the default is 43. That number is just an example, as the package manager for each distribution of PostgreSQL has discretion about the initial setting. Your mileage **will** vary.Changing this parameter requires a restart, so you will need to make a judgment call about how many concurrent processes you expect to kick off. Add that to this base number and restart your system. Of course, a reasonable number has been added for you in Timescale. Remember the CPU and memory limitations while you are making this adjustment.What to Do With A PostgreSQL Job Scheduler: A Few IdeasThe original reasons for creating this scheduler involve building out-of-the-box features involving data management. That includescompression,continuous aggregates,retention policy implementation,downsampling, andbackfilling.You may want to use this for event notifications, sending an email, clustered index maintenance, partition creation, pruning, archiving, refreshing materialized views, or summarizing data somewhere to avoid the need for triggers. These are just a few of the obvious ideas that jump into my consciousness. You can literally do anything that the operating system allows.WhatNotto DoThis would be a bad place to gum up the locking tables. That is, be sure that whatever you do here is done in a concurrent manner.REFRESH INDEX CONCURRENTLYis better thanDROP/CREATE INDEX.REFRESH MATERIALIZED VIEW CONCURRENTLYis better thanREFRESH MATERIALIZED VIEW. You get it. UseCONCURRENTLY, or design concurrently. Better yet, do things in a tiny atomic way that takes little time anyway.Long-running transactions that create a lot of locks will interfere with the background writer, the planner, and the vacuum processes. If you crank up too many concurrent processes, you may also run out of memory. Please try to schedule everything to run in series. You’ll thank me later.Well Wishes to the Newly Crowned EmperorNow you have the power to do anything your little heart desires in the background of PostgreSQL without having any external dependencies. We hope you feel empowered, awed, and a little bit special. We also hope you will use your new powers for good!Try the Updated Job SchedulerThe job scheduler is available in TimescaleDB 2.9.1 and beyond. If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation. If you are using theTimescale platform, upgrades are automatic, meaning that you already have the scheduler at your fingertips.Keep LearningIf this article has inspired you to keep going with your PostgreSQL hacking,check out our collection of articles on PostgreSQL fine tuning.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/the-postgresql-job-scheduler-you-always-wanted-but-be-careful-what-you-ask-for/
2022-09-29T12:57:28.000Z,Nightmares of Time Zone Downsampling: Why I’m Excited About the New time_bucket Capabilities in TimescaleDB,"Time-series data is generally collected at a much higher rate than it will be displayed. That’s why we fall back on downsampling or rolling up the data into lower granularities (or time buckets), a technique that helps you manage the endless flows of time-series data you’re collecting. However, when resorting to downsampling data collected at high frequency, there is one important element to keep in mind: the user’s time zone.Before I joined Timescale, I had the same issue in my own startup, as most customers were located in Central Europe (Europe/Berlin, to be precise) and not in the UTC+0 time zone. I can imagine that a lot of you are struggling with this too. If your project requires exact daily overviews and you need to define what midnight means in your customer’s time zone, I know you feel this pain!Trust me, we are not alone. The support for time zones intime_bucketwas one of the most requested (and upvoted) features in theTimescaleDB GitHub repoand in polls conducted inTimescale Community Slack. But now, the days of hoop-jumping to fix this issue are over: in the last few months—and with the help of its community—Timescale re-implemented thetime_bucketfunctionalitywith full time zone support, which is now merged into the latest TimescaleDB 2.8 release.Support for months, years, and time zones was the most requested TimescaleDB feature to dateWhy is this such a big deal? Because it will save you time, no matter what time zone you or your customers are in.Time Sucks, time_bucket, and Time ZonesAs I shared earlier, while at my own startup, we had to force TimescaleDB to maketime_bucket“support” time zones. It was way too ugly, though.The gist is to force the inputtimestamptzto be converted intotimestampwithout time zone information (but adjusting it accordingly to the requested time zone). Then you need to pass it to thetime_bucketfunction and have it bucketed. The resultingtimestamptzneeds to be ripped off of its time zone information again and finally re-interpreted as a timestamp inside the requested time zone. Sounds complicated?In terms of code, it looks like this:time_bucket('86400s', TIMEZONE('Europe/Berlin', created)::timestamptz)::TIMESTAMP WITHOUT TIME ZONE AT TIME ZONE 'Europe/Berlin' AS bucketAnyway, this workaround is now a thing of the past! The new TimescaleDB 2.8 release ships with a revamped version oftime_bucket,which offers full support for time zones. The new implementation was developed with the help of the community over the last couple of months (huge shout out to all of you who contributed!) and lived in Timescale’s experimental namespace astime_bucket_ng. That means that many of you are probably familiar with it already.The change in TimescaleDB 2.8 is that we graduated the new implementation out of thetimescale_experimentalnamespace and replaced the existingtime_bucketaltogether with the implementation oftime_bucket_ng.That said, we are happy to announce that we “fixed” the most requested feature to date.New Functionality and MigrationWhile the new functionality is rather simple, its impact is immense and simplifies analytical requests requiring time zone information.As I mentioned before, most of your customers are probably not located in a UTC time zone since there aren’t a lot of countries in that time zone anyway, or customers live in one of the many countries or economic areas spread across multiple time zones. That means you have to ensure that a user’s time zone is taken into account when rolling up over day, week, month, or year boundaries.From a migration standpoint, nothing will change for users not usingtimezone. The new implementation oftime_bucketbehaves like the old version, as long as you do not provide atimezoneparameter. That means the default behavior is still bucketing on UTC boundaries, and theoriginparameter is unchanged.When provided with atimezoneparameter, though,time_bucketwill adjust theoriginto the given time zone. That means that day boundaries, as well as months and years, are adjusted too. With that change, providing a time zone is as simple as the following query:SELECT
   time_bucket('1 day', created, 'Europe/Berlin') AS bucket,
   avg(value)
FROM metrics
GROUP BY 1The result is time buckets adjusted to midnight in the Central European Time area (represented by theEurope/Berlintime zone identifier), including changes due to summer or winter (CET/CEST).When talking about the migration of existing code, all your current queries will work just as they did before the change. All parameter combinations oftime_bucketare available as implementations under the covers, and only extended versions of the method signatures are added to bring time zone support.Withtime_bucketgraduating from thetimescale_experimentalnamespace, the first major step is done. Thetime_bucket_gapfill_ngfunction is still experimental, and we will be adding time zone support totime_bucket_gapfillin the very near future.Getting Started With TimescaleDB 2.8Timescale is happy to release the new version oftime_bucketafter months of experimental status, and we want to thank everyone in the community for their help and effort in testing this feature, making it production-ready, and giving feedback!Looking back, this functionality would’ve made things much simpler and more accurate in my startup (we never handled stuff like leap seconds), but you can and should use the new functionality immediately!There is, however, more to the release of TimescaleDB 2.8 than just a newtime_bucket. Other features include experimental support for thenew policy management for continuous aggregates, and speed improvements for distributed hypertables using theCOPYprotocol forSELECTs. For more details on these changes, see What’s New in TimescaleDB 2.8 (for a complete list of all changes, see theRelease Notes.)If you are using Timescale, upgrades are automatic, and you’ll be upgraded to TimescaleDB 2.8 in your next maintenance window.New to Timescale?Start a free 30-day trial, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/nightmares-of-time-zone-downsampling-why-im-excited-about-the-new-time_bucket-capabilities-in-timescaledb/
2023-03-28T13:01:24.000Z,"Scaling PostgreSQL With Amazon S3: An Object Storage for Low-Cost, Infinite Database Scalability","Last November, we announced that we were building an integrated object storage layer to scale PostgreSQL further in a cost-effective way.Today, we’re happy to announce that data tiering is available in Early Access for all Timescale customers.By running a simple command on your hypertable (add_tiering_policy), you can automatically tier older data to a low-cost, infinite storage layer built on Amazon S3. Yet the data still remains fully queryable from within your database, and this tiering is transparent to your application.This storage layer is an integral part of your database. Your Timescale hypertables now seamlessly stretch across our regular and object storage, boosting performance for your most recent data and lowering storage costs for older data. There are no limitations to the data volume you can tier to object storage, and you’ll be charged only for what youstore—no extra charge per query or data read, and no hidden costs.We're also rolling out a special offer to celebrate this awesome achievement (we can’t wait for you to try this!).During the Early Access period, this feature will be entirely free for all Timescale customers, meaning that you won’t be charged for the volume of data tiered to the object store during that time. We’ll disclose our final pricing for tiered data in the following weeks, but don’t worry—you won’t lose what you saved:  the object storage will be priced roughly 10x cheaper than our regular storage. Stay tuned for details.This feature is a step forward to solving a pressing problem we keep hearing from developers:it is simply too expensive and difficult to scale PostgreSQL databases in AWS.Products like Amazon RDS for PostgreSQL and Amazon Aurora work great for small deployments, but once the project grows, databases in RDS and Aurora start getting prohibitively expensive. By allowing you to tier data to object storagewithout leaving Timescale, we avoid the need to remove data from your database in an attempt to lower costs or to avoid hard limits related to disk capacity.Instead,you now have an easy path for scaling sustainably and within budget without terabyte limitations, paying only for what you use.""We perform a lot of analysis on market data, and the sheer volume of data we need to store makes a normal disk-based database solution unfeasible (it's just too expensive).Timescale’s data tiering seamlessly allows us to access large volumes of data on S3. This is a great solution to store large volumes of historical data and perform post-analysis. Without this, we'd be forced to develop a solution in-house."" (Chief Technology Officer at a Proprietary Digital Assets Trading Company)With the object store integrated into your database, Timescale hypertables stretch effortlessly across multiple cloud-native storage layers. This architecture gives you a seamless querying experience—even when data is tiered, you can still query it within the database via standard SQL, just like in TimescaleDB and PostgreSQL. Everything “just works”: predicates, filters, JOINs, common table expressions (CTEs), windowing, andhyperfunctions.Data tiering is available for all TimescaleDB services in Timescale: you can start tiering data immediately (free trial included). Check out our documentation for instructions on enabling your first tiering policy, or see the video below.""RDS Pricing Is Too Steep: Why PostgreSQL Needs an S3 Object Store""Many of our customers come from PostgreSQL-compatible products, including Amazon RDS for PostgreSQL, Amazon Aurora, and Heroku. Even when these customers’ use cases vary across the board (from finance and IoT to energy), the story they tell us is consistently the same:At the start of their project, developers choose PostgreSQL due to its reliability, ease of use, and broad compatibility. As their first option, they often pick RDS PostgreSQL (the path of least resistance in AWS) or Heroku (which has an appealing price point for small deployments).Everything works great for a while, but as the project grows, so does the volume of data. That’s when the queries start slowing down.As time goes by and more data gets ingested, the problem becomes increasingly critical. The database starts holding the application hostage. The team works hard to optimize the database, but the performance improvements are only temporary.At this point, the teams running on RDS or Heroku sometimes choose to move their workloads to Amazon Aurora, which promises better performance than RDS. While the team hopes the move will solve their sluggish queries, another problem arises when they switch to Aurora: untenable, painfully high costs.Aurora’s billing proves to be unpredictable, with costs soaring much higher than anticipated (hello, I/O), and as the data volume increases, the problem only gets worse. There’s no ceiling for this ever-growing database bill, and growth quickly seems unsustainable.By moving to Timescale, teams experiencing these issues solve their performance and cost problems. With features like hypertables and continuous aggregates,Timescale delivers up to 350x faster queries and 44 % faster ingestion than RDS PostgreSQL. With such performance improvements, Timescale users can use a smaller compute footprint to accomplish similar workloads, leading to significant savings. Unlike Aurora,Timescale’s pricingis transparent and predictable. And with Timescale, customers can alreadyreduce their storage footprint by more than 95 % via columnar compression, saving money and further improving query performance.And yet, we knew we could do more by establishing a direct connection from Postgres to S3. We wanted to take advantage of the lower cost and reliability of Amazon S3 to offer a cheaper storage option to PostgreSQL users in AWS,adding one order of magnitude more savings to those already offered via our native columnar compression.Imagine we were engineers scaling a data-centric application—this is what we would build for our own use. We did it, so you didn’t have to.Infinite, Low-Cost Database Scalability for PostgreSQLAll data inserted into Timescale is initially written into our faster storage layer built on the latest-generation, IO-optimized EBS. Using faster disks for your most recent data will bring you top insert and query performance for your most recent values—a usage pattern that fits well for time series, events, and other analytics use cases. Once your data gets older (and mostly immutable), you can tier it to the object store automatically by defining a time-based policy. And it’sreallyeasy.# Create a tiering policy for data older than two weeks
SELECT add_tiering_policy ('metrics', INTERVAL '2 weeks');Hypertables now transparently stretch across two different storage layers, enabling fast performance for your most recent data and affordable, infinite storage for your older dataBy building this, we’re providing a low-cost alternative for scaling your PostgreSQL databases in AWS. This is our special offer during the Early Access period:You won’t be charged for the data in object storage. For the next 1-2 months, data tiering will be free for all users.You won’t stop saving once the free period is over. The object storage will be roughly 10x cheaper than our regular storage.There are no limitations to the volume of data you can tier to object storage, and you will be charged only per gigabyte—no extra charges per query or other hidden costs.A Postgres Object Store Built on Amazon S3: Much More Than an External ArchiveThis object store is much more than an external bucket to archive your data: it’s an integral part of your database. When tiering data, your database will remain fully aware of all the semantics and metadata. You can keep querying as usual with standard SQL.With Timescale’s data tiering, all data tiered to S3 is in a compressed columnar format (specifically,Apache Parquet).  When data is tiered to S3 based on its age, chunks stored in Timescale’s native internal database format (typically itself in ournative columnar compressionstored in a Postgres-compatible data table) are asynchronously converted to Parquet format and stored in S3.  These tables remain fully accessible throughout the tiering process, and various mechanisms ensure that they are durably stored to S3 before transactionally removed from standard storage.When you run your SQL query, it will pull data from the disk storage, object storage, or both as required. And to avoid processing chunks falling outside the query’s time window, we perform “chunk exclusion” to our query those chunks minimally required to satisfy a query.Further, the database is selective in what data is read from S3 to improve query performance; it stores various forms of metadata to build a  “map” of row groups and columnar offsets within the S3 object.  If your query only touches a range of rows and few of the columns, only that subset of the data is read from the tiered S3 object. The result? Less data to fetch and thus faster queries.And when we say transparent, we mean transparent. Timescale supports arbitrarily complex queries across its standard and tiered data, including complex predicates, JOINs, CTEs, windowing, hyperfunctions, and more.In the example query below (with an EXPLAIN clause), you can see how the query plan includes a `Foreign Scan` when the database is accessing data from S3. In this example, three chunks were read from standard storage, and five chunks were read from object storage.EXPLAIN 
SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;


QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
    -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text)How to Get StartedStart slashing your storage bill today!Data tiering is available in Early Access for all Timescale customers. To activate it, simply navigate to your service’s Overview page and press “Enable data tiering.” For detailed instructions, see our documentation or watch our demo video below:Haven’t tried Timescale yet?Start a Timescale trial and get full access to the platform for 30 days, no credit card required.Data tiering is also available for trial services: start experimenting now!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/scaling-postgresql-with-amazon-s3-an-object-storage-for-low-cost-infinite-database-scalability/
2022-03-28T13:06:01.000Z,High Availability for Your Production Environments: Introducing Database Replication in Timescale,"Earlier this year, we kicked off the Year of the Tiger 🐯🦄 by announcing our$110M Series C fundingto build the future of data for developers worldwide. This new funding helps us accelerate the delivery of features that help our customers build best-in-class data-driven applications. This week, we’re excited to continue this momentum with #AlwaysBeLaunching (#ABL): MOAR Edition—a week full of exciting new features for Timescale, bringing you MOAR features that makeTimescaleeven MOAR worry-free, scalable, and flexible!To start, we’re releasing early access to a highly requested feature:database replication in Timescale.And throughout the rest of this #ABL Cloud Week, we’ll release features that give you MOAR collaboration, visibility, performance, and regions in Timescale.One of our guiding principles as a company is that boring is awesome and thatyour database should be boring: we believe that you should be able to focus on your applications rather than on the infrastructure on which they’re running. Replicas is a feature commonly available in most databases, and today we’re surfacing that functionality in an easy-to-use experience within Timescale. As with all things Timescale, we build on top of the proven functionality available in PostgreSQL, the foundation of TimescaleDB.Database replication in Timescale is as easy as pressing a button. By enabling a replica, you will increase the availability of your data, assuring that you will only experience a few seconds of downtime if your database fails—but that’s not all. In Timescale, enabling replicas can also improve performance, as you can easily direct your heavy read queries to the replica, which frees up resources in your main database for higher ingest rates, more advanced continuous aggregates, or additional read queries.The entire database replication process (including creating a replica, checking out its status, retrieving its URL, and deleting your replica) can be done easily and transparently from the Console UI. And if your database happens to fail – for example, because the underlying AWS server instance becomes unavailable – all the recovery processes are done automatically. Timescale will shift responsibilities from the former failed primary to the replica, which will then be elevated to start accepting write operations as a new primary without any action needed from you. Your operations will just keep on running while Timescale does this heavy lifting under the hood.Read on to learn more about replicas in Timescale, and how you can use database replication to increase data availability and liberate load in your Timescale database.If you’re new to Timescale,try it for free(100% free for 30 days, no credit card required). Once you’re up and running, join our community of 8,000+ developers passionate about TimescaleDB, PostgreSQL, time-series data, and all its applications! You can find us in theTimescale Community Forumandour Community Slack.A huge “thank you” to the teams of engineers and designers that made all the features we’re releasing during the #AlwaysBeLaunching a reality! 🙏Replicas in TimescaleYour database should be worry-free: you shouldn't have to think about it, especially if there’s a problem. As any database operator knows, failures occur. But a modern database platform should automatically recover and re-establish service, as soon as issues arise. Our commitment is to build a dependable, highly-available database that you can count on. Boring is awesome when the platform prevents you from getting paged at 3 am.Many architectural aspects of Timescale are intentionally aligned with this high availability goal. For example, Timescale’s decoupled compute and storage is not only great for price optimization but also for high availability. In the face of failures, Timescale automatically spins up a new compute node and reconnects it to the existing decoupled database storage, which itself is independently replicated for high availability and durability. Indeed, even without a replica enabled, this cloud-native architecture can provide a full recovery for many types of failures within 30-60 seconds, with more severe physical server failures often taking no more than several minutes of downtime to recover your database.Further, incremental backups are taken continuously on Timescale for all your services (and stored separately across multiple cloud availability zones), allowing your database to be restored to any point in time from the past week or more. And Timescalecontinuously smoke tests and validates all backupsto ensure they are ready to go at a moment’s notice.However, many customers run even their most critical services on Timescale, where they need almost zero downtime—even in the case of unexpected and severe hardware failures. Indeed, Timescale powers many customer-facing applications, where downtime comes with important consequences for the business: application dashboards stop responding, assembly lines are no longer monitored, IoT sensors can no longer push measurements, critical business data can be lost, and more. Database replication provides that extra layer of availability (and assurance) these customers need.Apart from decreasing downtime, Timescale replicas have another advantage: they can also help you ease the load from your primary database. If you are operating a service subjected to heavy read analytical queries (e.g., if you’re using tools like Tableau or populating complex Grafana dashboards), you can send such read queries to the replica instead of to your primary database, liberating its capacity for writes and improving performance. This makes your replica useful even in the absence of failure.And leveraging this functionality is as easy as using a separate database connection string that Timescale makes available: one service URL for your (current) primary, and the platform transparently re-assigns this connection string to a replica if that replica takes over and a second service URL that maps to your read-only replica.If you are already a Timescale user, you can immediately set up a database replica in your new and existing services. Enabling your first replica is as simple as this:Select the service you’d like to replicate.Under “Operations,” select “Replication” on the left menu.To enable your replica, click on “Add a replica.”That’s it! 🔥If a service has a replica enabled, it will show under Operations -> Replication. (Take into account that the replica won’t show in your Services screen as aseparate database service,as it is not an independent service.)You can enable a replica in Operations -> Replication. Once created, your replica status will show on this page.If you want to direct some of your read queries to the replica, go to your service's “Overview” page. Under “Connection info,” you will see a drop-down menu allowing you to choose between “Primary” and “Replica.” To connect to your replica, you can simply select “Replica” and use the corresponding service URL.To connect to your replica, select Replica on your Overview page and use the corresponding service URL.We’re releasing database replication with an early access label, meaning that this feature is still in active development. We will add new functionality to replicas in the very near future: at the end of this post, you will find a detailed list of everything we’re actively working on regarding replicas.You’ll hear from us again soon!How Replicas WorkReplicas are duplicates of your main database, which in this context is called “primary.” When you enable a replica, it will stay up-to-date as new data is added, updated, or deleted from your primary database. This is a major difference between a replica and a fork,which Timescale also supports: a fork is a snapshot of your database in a particular moment in time, but once created, it is independent of your primary. After forking, the data in your forked service won’t reflect the changes in the primary.PostgreSQL (and thus TimescaleDB) offersseveral methods for replication. However, setting up replication for a self-hosted database is a difficult task that includes many steps—from choosing which options suit you best, to actually spinning up a new server and adjusting configuration files, to tweaking configurations, and to building a full infrastructure that monitors the health of your primary and automatically failing-over to a replica when needed.  All the while avoiding “split-brain” scenarios in which two separate services both believe they are primaries, leading to data inconsistency issues.Timescale automates the process for you: we do the hard work so you don’t have to.As we’ve seen in the previous section, adding a replica to your service is extremely simple. But as a more technical deep dive for the interested reader, the following paragraphs cover three design choices we’ve taken for replicas: (i) the choice of asynchronous commits, (ii) their ability to act as hot standbys, and (iii) their use of streaming replication.Timescale replicas are asynchronousThe primary database will commit a transaction as soon as they are applied to its local database, at which point it responds to a requesting client with success.  In particular, it does not wait until the transaction is replicated and remotely committed by the replica (as would be the case with synchronous replicas). Instead, the transaction is asynchronously replicated to the replica by the primary shipping its Write-Ahead Log (WAL) files, hence the “quasi-real-time” synchronicity between primary and replica.We chose this design pattern for two important reasons. The first one, perhaps surprisingly, is high availability: with a single synchronous replica, the database service would stop accepting new writes if the replica fails, even if the primary remains available.  The second is performance, as database writes are both lower latency (no round trip to the replica before responding) and can achieve higher throughput.  And that’s important for time-series use cases, where ingest rates are often quite high and can be bursty as well.In the future, when we add support for multiple replicas, we plan to introduce the ability to configure quorum synchronous replication, where a transaction is committed once written to at least some replicas, but not necessarily all. This addresses one tradeoff with asynchronous replication, where a primary failure may lead to the loss of a few of the latest transactions that have yet to be streamed to any replicas.Timescale replicas act like hot standbysAwarm standbymeans that the replica (standby) is ready to take over operations as soon as the primary fails, as opposed to a “cold standby” which might take a while to restore before it can begin processing requests.  This is closely related to the high availability mentioned earlier. The process of the standby/replica becoming the primary is called failover, which is covered more below. Since Timescale replicas are also read replicas – i.e., they can also be used for read-only queries – they are consideredhot standbysinstead of justwarmstandbys.As we will talk about in later sections, allowing you to read from your replicas gives you the option to direct some of your read-only workloads to your replica, freeing capacity in your primary. This means that in Timescale, you will not only have a replica ready to take over at any moment if the primary happens to fail, but you can also get value from it beyond availability, even if there’s no failure.Timescale replicas use streaming replicationStreaming replication helps ensure there is little chance of data loss during a failover event.  Streaming replication refers to how the database’s Write-Ahead Log – which records all transactions on the primary – is shipped from the primary to the replica. One common approach for shipping this WAL is aptly-named “log shipping.” Typically, log shipping is performed on a file-by-file basis, i.e., one WAL segment of 16MB at a time. So, these files aren’t shipped until they reach 16MB or hit a timeout.The implication of log shipping, however, is that if a failure occurs, any unshipped WAL is lost. Instead of file-based log shipping, Timescale uses streaming replication to minimize potential loss. This means that individual records in the WAL are streamed to the replica as soon as they are written by the primary rather than waiting to ship as an entire segment. This method minimizes the potential data loss to the gap between a transaction committing and the corresponding WAL generation.Enabling Replicas for High AvailabilityEven without a replica enabled, Timescale has a range ofautomated backup and restore mechanismsthat protect your data in case of failure. For example, the most common type of failure in a managed database service is a compute node failure; in Timescale, it often takes only tens of seconds to recover from such a failure, as we are able to spin up a new compute node and reattach your storage to it. In the much rarer case, in which a failure affects your (replicated) storage, Timescale automatically restores your data from backup, at a rate of roughly 10 GB per minute.For some use cases, the potential level of downtime associated with this recovery process is completely acceptable; for those customer-facing applications that require minimal downtime, replicas will provide the extra layer of availability they need.The recovery process through replicas is summarized in the figures below. In a normal operating state, the application is connected to the primary and optionally to its replica to scale read queries.  Timescale manages these connections through load balancers, defining the role for each node automatically.In a normal operating state, the application is connected to the primary and optionally to its replica. The load balancer handles the connection and defines the role for each node.The next figure illustrates a failover scenario. If the primary database fails, the platform automatically updates the roles, “promoting” the replica to the primary role, with the primary load balancer redirecting traffic to the new primary. When the failed node either recovers or a new node is spun up, it assumes the replica role. The promoted node remains the primary, streaming WAL to its new replica.When the primary database fails, the platform updates the roles, “promoting” the replica to the primary role, with the primary load balancer redirecting traffic to the new primary. In the meantime, the system begins the recovery of the failed node.When the failed node either recovers or a new node is spun up, it assumes the replica role. The promoted node remains the primary, streaming WAL to its new replica.When the failed node recovers or a new node is created, it assumes the replica role. The previously promoted node remains the primary, streaming WAL to its replica.On top of increasing the availability of your database in case of failure, replicas also essentially eliminate the downtime associated with upgrades, including database, image, or node maintenance upgrades. Without a replica, these upgrades usually imply 30 to 60 seconds of downtime. With a replica, this is reduced to about a second (just the time to failover). In this case, when the upgrade process starts, your system will switch over to the replica, which now becomes the primary. Once the upgrade is completed in the now-replica-formerly-primary, the system switches back so it can subsequently upgrade the other node. (And on occasions the replica is upgraded first, in which case only one failover will be necessary.)Read Replicas: Enabling Replicas for Load ReductionTimescale's replicas act as “hot standbys” and thus also double as read replicas: when replicas are enabled, your read queries can be sent to the replica instead of the primary.The main advantages of read replicas are related to scalability. By allowing your replica to handle all of the read load, your primary instance only has to handle writes or other maintenance tasks that generate writes, such as TimescaleDB’s continuous aggregates. Using read replicas would result in higher throughput for writes and faster execution times on analytical reads, plus a less strained primary instance.For example, read replicas can be particularly useful if you have many Grafana dashboards connecting to your service. Since visualizations don’t need perfectly real-time data – that is, using data that’s a few seconds old is often more than acceptable – the replica can be used to power these dashboards without consuming resources on the primary. Plus, with this setup, data analysts can work with up-to-date production data without worrying about accidentally impacting the database operations, such as with more ad-hoc data science queries.Another benefit of using read replicas is to limit the number of applications with write access to your data. Since the entire replica database is read-only, any connection, even those with roles that would have write privileges in the primary, cannot write data to the replica. This can serve to easily isolate applications that should have read/write access from those that only need read access, which is always a good security practice.  Database roles should certainly also be used to ensure “least privilege,” but a bit of redundancy and “defense in depth” doesn’t hurt.Coming SoonToday, we’re releasing database replication under an “early access” label, meaning that this feature is still in active development. We’ll be continuing to develop capabilities around database replication in Timescale, including:Multiple replicas per database serviceGreater flexibility around synchronous vs. asynchronous replicasReplicas in different AWS regionsReplicas in multi-nodedatabaseservicesSo keep an eye out – MOAR replication options coming soon!Get StartedTimescale’s new database replication provides you with increased high availability and fault tolerance for your important database services. In addition, it allows you to scale your read workloads and better isolate your primary database for writes. Check out our documentation for more information on how to use this database replication in Timescale.Replicas are immediately available for Timescale users. If you want to try Timescale,you can create a free account to get started—it’s 100% free for 30 days, without a credit card required. And if you have any questions, you can find us in ourCommunity Slackand also in theCommunity Forum.And if this sounds like the type of technical challenges you enjoy working on:We’re hiring. Fully remote and globally distributed.So let’s kick off the first Cloud Week 2022 with MOAR availability. And stay tuned – many more exciting Timescale capabilities to come!🐯🚀Eon, our Timescale mascot, roars “MOOARRRR” in honor of #AlwaysBeLaunching!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/high-availability-for-your-production-environments-introducing-database-replication-in-timescale-cloud/
2022-10-25T15:17:33.000Z,3 Big Reasons Why You Should Upgrade to the New Continuous Aggregates,"Continuous aggregates(or “caggs,” as we like to call them) were developed by Timescale to solve one of the biggest challenges application developers and data engineers faced: aggregating massive amounts of data efficiently without always having to query billions (or trillions) of raw data rows.Continuous aggregates solve this problem by usingPostgreSQL materialized viewsto continuously and incrementally refresh a query in the background, so that when you run said query, the only data that needs to be computed is the one that changed, instead of the entire dataset.In theprevious versionof TimescaleDB, we upgraded continuous aggregates, making them up to44,000x fasterin some queries than in previous versions.Sounds impressive, right? But it gets even better.Now, with TimescaleDB 2.8.1, you don’t have to worry aboutmigrating from the old continuous aggregates to the new. Say hello to our frictionless migration, an in-place upgrade that avoids disrupting queries over continuous aggregates in applications and dashboards and every time the data is not in the original hypertable.Also, we should mention that the migration to the new version of continuous aggregates is not available in TimescaleDB versions before 2.8.1. If you are planning to migrate a continuous aggregate to the new format, make sure to upgrade to TimescaleDB 2.8.1 first.Now that you know that this is going to be a fuss-free move, here are three reasons why you should upgrade to the new version of continuous aggregates.1. Even Faster QueriesEnjoy faster queries for the following common queries:Counting rows by withCOUNT(*) FROMis now up to 15x faster.Counting the number of records where the aggregate value is within a certain range, usingSELECT COUNT(*)...based on the value of a column is now up to 1,336x faster.Querying data within a range of time to get the top rows withORDER BYis now up to 15x to 44,000x faster.Querying data withtime_bucket()is now up to 160x faster.Querying data withFILTERis now 5x faster.Typical candlestick aggregates are nearly 2,800x faster to query than in previous versions of continuous aggregates.Querying using theHAVINGclause is now 4.8x faster.Fewer data rows mean faster queries. Continuous aggregates increase query speed by decreasing the overall row count by at least 20x. The new continuous aggregates reduce the previously required second grouping withSELECT COUNT(*) FROM…. That means TimescaleDB can query the data normally, leading to faster queries.Another performance improvement shows us 1,336x faster queries when counting the number of records where the aggregate value is within a certain range. The new continuous aggregates allow TimescaleDB to search for values directly and add an index to two specific columns for increased query performance.When querying data within a range of time to get the top rows, performance improved 45,000x. How do continuous aggregates achieve such an impressive boost? By only searching through the materialized data. They also allow hyperfunctions to re-aggregate into higher time buckets.2. Reduced Storage CostsContinuous aggregates were previously incompatible with common SQL features, such asFILTER, requiring workarounds that used more rows. Now, the aggregated data can be stored using aFILTERclause to achieve the results without additional effort or rows. This saves storage, cuts rows by 50 percent in our sample data set, and saves CPU to finalize theCOUNT(*)and then filter results each time.3. Greater Query FlexibilityThe new continuous aggregates also let you apply theHAVINGclause at materialization time, only storing rows that meet certain criteria and thus reducing the overall data footprint.The new ability to use common SQL features provides not only storage savings but more flexibility. These are the aggregate queries now compatible with continuous aggregates:DISTINCT,FILTER,FILTERin theHAVINGclause, aggregates without combine function, ordered-set aggregates, and hypothetical-set aggregates.Try Today on TimescaleNew to Timescale?Start a free 30-day trial, no credit card required, and get your TimescaleDB 2.8.1 database journey started in five minutes.If you are using Timescale, upgrades are automatic, and you’ll be upgraded to TimescaleDB 2.8.1 in your next maintenance window.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/3-big-reasons-why-you-should-upgrade-to-the-new-continuous-aggregates/
2022-02-23T14:23:08.000Z,Increase Your Storage Savings With TimescaleDB 2.6: Introducing Compression for Continuous Aggregates,"Yesterday, we announced thatwe raised $110 million in our Series C. 🐯🦄🚀 Today, we keep celebrating the big news Timescale-style: with an #AlwaysBeLaunching spirit! We're excited to announce the release of TimescaleDB 2.6, a version that comes with new features highly requested by our community - most notablycompression for continuous aggregates. 🔥 TimescaleDB 2.6 also includes theexperimentalsupport fortimezones in continuous aggregates.We love building in public. We are firm believers in the value of user feedback, as there’s no better way to improve your product than hearing from those who use it daily. We read all the thoughts and comments you share in our Community Slack, and we pay close attention toyour feature requests in GitHub. When features get upvoted, it helps us prioritize what to work on next.With today’s release, we are proud to bring you a top requested feature:the support for compression in continuous aggregates. Originally, we envisioned that compression would mostly be necessary over raw data, as continuous aggregates by themselves help downsample datasets considerably. But TimescaleDB users operate at such a scale thatthey requested compression also for their continuous aggregatesto save even more disk space.Theissuethat originated the support for compression in continuous aggregatesContinuous aggregatesspeed up aggregate queries over large volumes. You can think of them as a more powerful version of PostgreSQL materialized views, as they allow you to materialize your data while your view getsautomatically and incrementally refreshed in the background. But continuous aggregates are also very useful for something else: downsampling. Indeed, another property of continuous aggregates is that you can keep them around even when the data from the underlying hypertable has been dropped. This allows you to reduce the granularity of your data once it reaches a certain age, liberating space while still enabling long-term analytics.The ability to compress continuous aggregates takes this one step further. Starting with TimescaleDB 2.6, you can apply TimescaleDB’snative columnar compressionto your continuous aggregates, freeing even more disk space. And by combiningcompression policies(which automatically compress data after a certain period of time) withdata retention policies, you can automatically set up a downsampling strategy for your older data.Figure describing the downsampling process through continuous aggregates and compression. Notice the relationship between the refresh policy, compression policy, and data retention policy.TimescaleDB 2.6 also comes with another highly requested feature by the community:you're now able to create continuous aggregates with monthly buckets and/or timezonesusingtime_bucket_ng. (Note that this is anexperimentalfeature.)time_bucket_ngis the “new generation” of ourtime_buckethyperfunction, used for bucketing and analyzing data for arbitrary time intervals in SQL. You can think oftime_bucketas a more powerful version of thePostgreSQLdate_truncfunction, allowing for arbitrary time intervals rather than the standard day, minute, hour provided bydate_trunc. Buttime_bucketdoesn’t yet support buckets by months, years, or timezones;time_bucket_ngexpands the capabilities oftime_bucketby including these features in our experimental schema. (We first introduced this featurethanks to an issue from the community.)We’re eager to hear how this works for you. Our main goal with experimental features is to get as much input as possible - please, if you see something that needs improvements,tell us in GitHub!Theissuethat originated the support for timezones in continuous aggregates.​​TimescaleDB 2.6 is available today. If you are already a TimescaleDB user,check out our docs for instructions on how to upgrade. If you are using Timescale Cloud, upgrades are automatic, and no further action is required from your side (you can also start a free 30-day trial, no credit card required).If you are new to Timescale and you want to learn more about continuous aggregates, compression, andtime_bucket_ng, keep reading!Once you’re using TimescaleDB, join our community. You can ask us questions in ourCommunity Slackor in our brand newCommunity Forum, a better home for long-form discussions. We’ll be more than happy to solve any doubts you may have about compression and continuous aggregates, TimescaleDB 2.6, or any other topic. And if you share our mission of helping developers worldwide, we arehiring broadly across many roles!Before moving on, a huge thank you to the team of engineers (and the entire team of reviewers and testers) that made this release possible. This includes the community members who helped us prioritize, develop, and test these features. As we’ve said before, none of this would be possible without your requests, your upvotes, and your feedback.Please, keep ‘em coming!The power of continuous aggregatesContinuous aggregates in TimescaleDBare a more powerful version ofPostgreSQL materialized views. For example, continuous aggregates are incrementally updated with a built-in refresh policy, so they stay up-to-date as new data is added. When querying a continuous aggregate,the query engine will combine the data that is already pre-computed in the materialized view with the newest raw data in the underlying hypertable. In other words, you will always see up-to-date results.TimescaleDB uses internal invalidation records to determine which data has been changed or added in the underlying hypertable since the last refresh; when you refresh your continuous aggregate,only the new or updated data is computed.This means that TimescaleDB doesn’t need to look at your whole table every time you do a refresh, saving tons of computation resources, speeding up the incremental maintenance of continuous aggregate, and allowing you to get faster results in your aggregate queries.This also implies that you can also use continuous aggregates for downsampling. Differently than with materialized views, you are able to retain the continuous aggregate even after the original raw data has been dropped. To reduce the granularity of your time-series data, you can simply define a continuous aggregate and delete your original data once it gets old - we will be demonstrating this process later in this blog post.Compression in TimescaleDBTogether with continuous aggregates,compressionis another oft-used feature of TimescaleDB. By defining acompression policy, TimescaleDB allows you to compress all data stored in a specific hypertable that is older than a specific period of time - let’s say, 7 days old. In this example, you would keep your last week’s worth of data uncompressed, which would allow you to write data into your database at very high rates, giving you optimal query performance for your shallow-and-wide queries as well. Once your data gets old enough (after 7 days), the compression policy would kick in, automatically compressing your data.TimescaleDB is able to give you high compression rates by deployingbest-in-class compression algorithmsalong with anovel hybrid row/columnar storage design. Once a chunk inside your hypertable becomes old enough, TimescaleDB compresses it by storing the columns into an array. In other words: your more recent chunks (in the previous example, all chunks newer than 7 days) will be stored in TimescaleDB as relational, row-based partitions. Once compressed, your chunks will be a columnar store.An example chunk before compression vs after compression. After compression, the data previously stored in multiple rows is now stored as a single row, with the columns being stored as an array.✨ If you want to dive deeper into how compression works in TimescaleDB, check out the following videos:Compression 101 (part 1): learn all the fundamentals of compression and compression policies in TimescaleDB.Compression deep dive (part 2): expand your compression knowledge by getting familiar with more advanced functionality, like how to adequately use and configureSEGMENT BYcolumns.How to use continuous aggregates with compression for downsamplingAs we introduced earlier, continuous aggregates can help you reduce the granularity of your dataset, with the ultimate goal of saving some disk space.We explained previously how, in TimescaleDB, you are able to retain the continuous aggregate even after the original raw data has been dropped. So in order to considerably reduce your data size (and thus your storage costs) automatically and without losing the ability to do long term analytics on your older data, you can:Create a continuous aggregatecapturing the information that you most likely will like to see in your historical analytic queries.Define a refresh policy for your continuous aggregateso your continuous aggregate can stay up to date, periodically materializing your newest data.Enable compressionin the continuous aggregate.Add a compression policyto compress your chunks automatically once they’re older than a specific period of time.Create a data retention policyto automatically drop the raw data in the original hypertable once it gets older than a specific period of time.In this section, we’ll walk you through an example, using financial data fromAlpha Vantage. (If you want to also load this dataset into your TimescaleDB instance, we have astep-by-step guide published in our docs.)We will be using the following schema:CREATE TABLE public.stocks_intraday (
	""time"" timestamptz NOT NULL,
	symbol text NULL,
	price_open float8 NULL,
	price_high float8 NULL,
	price_low float8 NULL,
	price_close float8 NULL,
	trading_volume int4 null
	);
    
SELECT create_hypertable ('public.stocks_intraday', 'time');Create a continuous aggregate and set up continuous aggregate policyTo start, let’screate a continuous aggregateusing TimescaleDB’stime_bucket()function. We will record averages for the price at open, high, low, and close for each company symbol over a given day.We also will set up acontinuous aggregate policy, which will update and refresh data from the last four days. This policy will run once every hour:CREATE MATERIALIZED VIEW stock_intraday_daily
WITH (timescaledb.continuous, timescaledb.materialized_only = true) AS
SELECT 
time_bucket( interval '1 day', ""time"") AS bucket,
AVG(price_high) AS high,
AVG(price_open) AS open,
AVG(price_close) AS close,
AVG(price_low) AS low,
symbol 
FROM stocks_intraday si 
GROUP BY bucket, symbol;Enable and set up compression on your continuous aggregateNow that we have defined the continuous aggregate together with a refresh policy for it, we canenable compressionon this continuous aggregate, also setting up ourcompression policy. This compression policy will automatically compress all chunks older than 7 days:ALTER MATERIALIZED VIEW stock_intraday_daily SET (timescaledb.compress = true);

-- Set up compression policy
SELECT add_compression_policy('stock_intraday_daily', INTERVAL '7 days');Lastly, it is important to notice thatupdating data within a compressed chunk is not supported yet in TimescaleDB. This is relevant for correctly configuring compression policies in continuous aggregates: since refresh policies require chunks to be updated, we have to make sure that our recent chunks remain uncompressed. I.e., make sure you define your time intervals so that your continuous aggregate gets refreshed at a later date than when your compression policy is set.How compression affects storageSo we officially have a compressed continuous aggregate, but you may be wondering: how much of a difference does compression make on this continuous aggregate?In order to find out, let’s check out the stats on our compressed continuous aggregate. To do that, we first need to find the internal name for the materialized hypertable. We can do it by looking at our compressed tables:SELECT * FROM timescaledb_information.compression_settings;Then, we can use that to run thehypertable_compression_stats()command.(Note: The name of your materialized hypertable will most likely be different than the one shown below.)SELECT * FROM hypertable_compression_stats('_timescaledb_internal._materialized_hypertable_3');

Results:
—---------------------------------|--------
total_chunks                   | 7
number_compressed_chunks       | 6
before_compression_table_bytes | 7471104
before_compression_index_bytes | 2031616
before_compression_toast_bytes | 49152
before_compression_total_bytes | 9551872
after_compression_table_bytes  | 401408
after_compression_index_bytes  | 98304
after_compression_toast_bytes  | 3178496
after_compression_total_bytes  | 3678208
Node_nameFor this continuous aggregate, we got a compression rate of over 61%:Size of the continuous aggregate before and after compression (9.6 MB vs 3.7 MB)A 61% compression rate would imply a very nice boost in your storage savings. However, if you’re used to compression in TimescaleDB, you may be wondering: whyonly61%? Why cannot I getcompression rates over 90%as I commonly see with my hypertables?The reason behind this is that continuous aggregates store partial representation of the aggregates inbytea format, which is compressed using dictionary-based compression algorithms. So continuous aggregates cannot take advantage of other compression algorithms - at least, not yet. We’re experimenting with some concepts that may significantly increase the compression rates for continuous aggregates in the future. Stay tuned!Set up a data retention policy on your raw dataThrough the continuous aggregate we created earlier, we could effectively downsample our data. Our original dataset had one datapoint per minute; this is perfect for real-time monitoring, but a bit too heavy for the purpose of long-term analysis. In our continuous aggregate, we’re aggregating the data into 1 h buckets, which is a more manageable granularity if you’re planning to store this data long-term. So let’s save up some additional disk space by dropping the data in the underlying hypertable while keeping the continuous aggregate.To do so, we will define aretention policythat automatically deletes our raw data once it reaches a certain age. This age is completely up to you and your use case (the retention policy below will delete the data older than a month, for example). As a reminder, this policy refers only to the data in your original hypertable - you will still keep the data materialized in the continuous aggregate.SELECT add_retention_policy('stocks_intraday', INTERVAL '1 month');You asked, and we delivered: introducing timezones for continuous aggregatesAs we mentioned at the beginning of this post, TimescaleDB 2.6 introduces not only compression for continuous aggregates but also the possibility of using timezones in continuous aggregates,another highly requested feature by our community. Starting with TimescaleDB 2.6, you're able to create continuous aggregates with monthly buckets and/or timezones usingtime_bucket_ng.This is an experimental feature: please,test it and send us your feedback!The more feedback we get, the faster we will make this feature ready for production. 🔥time_bucket_ngis the “new generation” of ourtime_buckethyperfunction,  used for bucketing and analyzing data for arbitrary time intervals in SQL. You can think oftime_bucketas a more powerful version of thePostgreSQLdate_truncfunction, allowing for arbitrary time intervals rather than the standard day, minute, hour provided bydate_trunc.Buttime_bucketdoesn’t support time buckets by months, years, or timezones.time_bucket_ngexpandstime_bucketby including these features, giving users maximum flexibility in their queries. (We also first introducedtime_bucket_ngthanks to an issue from the community!)time_bucket_ngis still an experimental feature, being that it’s still being developed underour experimental schema. In Timescale, we like to bring constant value to our users by moving fast, but we also value stability - as we like to say, wemove fast without breaking things. Through our experimental schema, we’re allowed to release experimental features that, even if they’re still not ready for production, are ready to be widely tested. As we keep mentioning through this post, we love to get as much feedback from the community as possible: releasing features under experimental helps us build robust features, as by the time we “graduate” these features out of the experimental schema, we are sure that everything is stable.Starting with TimescaleDB 2.6, you can now use time buckets of months and years plus specify timezones in your continuous aggregates. For example, the continuous aggregate below tracks the temperature in Honolulu over monthly intervals.(Yes, we are dreaming of warmth 🏝 in this team!)CREATE TABLE conditions(
  day timestamptz NOT NULL,
  city text NOT NULL,
  temperature INT NOT NULL);

SELECT create_hypertable(
  'conditions', 'day',
  chunk_time_interval => INTERVAL '1 day'
);

INSERT INTO conditions (day, city, temperature) VALUES
  ('2021-06-14 00:00:00 HST', 'Honolulu', 26),
  ('2021-06-15 00:00:00 HST', 'Honolulu', 22),
  ('2021-06-16 00:00:00 HST', 'Honolulu', 24),
  ('2021-06-17 00:00:00 HST', 'Honolulu', 24),
  ('2021-06-18 00:00:00 HST', 'Honolulu', 27),
  ('2021-06-19 00:00:00 HST', 'Honolulu', 28),
  ('2021-06-20 00:00:00 HST', 'Honolulu', 30),
  ('2021-06-21 00:00:00 HST', 'Honolulu', 31),
  ('2021-06-22 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-23 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-24 00:00:00 HST', 'Honolulu', 34),
  ('2021-06-25 00:00:00 HST', 'Honolulu', 32),
  ('2021-06-26 00:00:00 HST', 'Honolulu', 32),
  ('2021-06-27 00:00:00 HST', 'Honolulu', 31);

CREATE MATERIALIZED VIEW conditions_summary
WITH (timescaledb.continuous) AS
SELECT city,
   timescaledb_experimental.time_bucket_ng('1 month', day, 'Pacific/Honolulu') AS bucket,
   MIN(temperature),
   MAX(temperature)
FROM conditions
GROUP BY city, bucket;

-- to_char() is used because timestamptz is displayed in the sesison timezone by default
-- alternatively you can use SET TIME ZONE 'Pacific/Honolulu';
SELECT city, to_char(bucket at time zone 'HST', 'YYYY-MM-DD HH24:MI:SS') as month, min, max
FROM conditions_summary
ORDER by month, city;

   city   |        month        | min | max
----------+---------------------+-----+-----
 Honolulu | 2021-06-01 00:00:00 |  22 |  34
(1 row)We’re eager to know how this feature is working for you so we can improve it. Please, reach out to us throughGitHub, ourCommunity Slack, or theTimescale Community Forum.And thank you again for your invaluable support!Get startedTimescaleDB 2.6 is already available for Timescale Cloud and self-managed TimescaleDB:If you are a Timescale Cloud user, you will be automatically upgraded to TimescaleDB 2.6during your next maintenance window. No action is required from your side. You can also create a free Timescale Cloud account to get a free 30-day trial, with no credit card required.If you are using TimescaleDB in your own instances,check out our docs for instructions on how to upgrade.If you are using Managed Service for TimescaleDB, TimescaleDB 2.6 will be available for you in the upcoming weeks.Once you’re using TimescaleDB, connect with us! You can find us in ourCommunity Slackand theTimescale Community Forum. We’ll be more than happy to answer any question on continuous aggregates, compression, TimescaleDB, PostgreSQL, or anything in between.And if you want to help us build and improve features like compression, continuous aggregates, andtime_bucket,we are hiring broadly across many roles! Join our global, fully remote team. 🌎Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/increase-your-storage-savings-with-timescaledb-2-6-introducing-compression-for-continuous-aggregates/
2022-06-22T13:00:24.000Z,How We Fixed Long-Running PostgreSQL now( ) Queries (and Made Them Lightning Fast),"It was just another regular Wednesday in our home offices when we received a question in theforum. A TimescaleDB user with dozens of tables of IoT data reported a slow degradation in query performance and a creeping server CPU usage. After struggling with the issue, they turned to our community for help.via GIPHYThat same question came up in our forum,Community Slack, andsupportmore often than we’d like. We could relate to this particular pain point because we also struggled with it in a partitioned vanilla PostgreSQL. After a closer look at the user’s query, we found the usual suspect: the issue of high planning time in the presence of many chunks—in Timescale slang, chunks are data partitions within a table—and in a query using a rather common function:now().Usually, the problem with these queries is that the chunk exclusion happens late. Chunk exclusion is what happens when some data partitions are not even considered during the query to speed up the process. The logic is simple: the fewer data a query has to go through, the faster it is.However, the problem is thatnow(),similarly to other stable functions in PostgreSQL, is not considered during plan-time chunk exclusion, those precious moments in which your machine is trying to find the quickest way to execute your query while excluding some of your data partitions to further speed up the process. So, your chunks are only excluded later, at execution time, which results in higher plan time—and yes, you guessed it—slower performance.Until now, every time this issue popped up, we knew what to do. We had written a wrapper function, marked as immutable, that would call thenow()function and whose only purpose was to add the immutable marking so that PostgreSQL would consider it earlier during plan-time chunk exclusion, thus improving query performance.Well, not anymore.Today, we’re announcing the optimization of thenow()function with the release of TimescaleDB 2.7, which solves this problem by natively performing as our previous workaround.In this blog post, we’ll look at the basics of thenow()function, explain how it works in vanilla PostgreSQL and our previous TimescaleDB version, and wrap everything up with a description of our optimization and performance comparison that will blow you away (all we can say for now is “more than 400 times faster”).via GIPHYIf you are already a TimescaleDB user,check out our docs for instructions on how to upgrade. If you are using Timescale, upgrades are automatic, so all you need to do is sit back and enjoy this very fast ride! (New to Timescale?You can start a free 30-day trial, no credit card required.)now( ) in Vanilla PostgreSQLQueries withnow()expressions are common in time-series data to retrieve readings of the last five minutes, three hours, three days, or other time intervals. In sum,now()is a functionthat returns the current time or, more accurately, the start time of the current transaction. These queries usually only need data from the most recent partition in a hypertable, also called chunk.A query to retrieve readings from the last five minutes could look like this:SELECT * FROM hypertable WHERE time > now() - interval ‘5 minutes’;To understand our users' slowdown, it’s vital to know that constraints in PostgreSQL can be constified at different stages in the planning process. The problem withnow()is that it can only be constified during execution because the planning and execution times may differ.Sincenow()is a stable function, it’s not considered for plan-time constraint exclusion; therefore, all chunks will have to be part of the planning process. For hypertables with many chunks, this query's total execution time is often dominated by planning time, resulting in poor query performance.If we dig a little deeper with the EXPLAIN output, we can see that all chunks of the hypertable are part of the plan, painfully increasing it.Append  (cost=0.00..1118.94 rows=1097 width=20)
   ->  Seq Scan on _hyper_3_38356_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38357_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38358_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38359_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38360_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())
   ->  Seq Scan on _hyper_3_38361_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())We had to do something to improve this, and so we did.now( ) in TimescaleDBAs proud builders on top of PostgreSQL, we wanted to come up with a solution. So in previous versions of TimescaleDB, we did not use thenow()expression for plan-time constraint exclusion.In turn, we implemented constraint exclusion at execution time in a bid to improve query performance. If you want to learn more about how we did this,check out this blog post, which offers a detailed behind-the-scenes explanation of what happens when you execute a query in PostgreSQL.While the resulting plan does look much slimmer than the original, all the chunks were still considered during planning and removed only during execution. So, even though the resulting plan looks very different (look at those 1,096 excluded chunks), the effort is very similar to the vanilla PostgreSQL plan.Custom Scan (ChunkAppend) on metrics1k  (cost=0.00..1113.45 rows=1097 width=20)
   Chunks excluded during startup: 1096
   ->  Seq Scan on _hyper_3_39453_chunk  (cost=0.00..1.01 rows=1 width=20)
         Filter: (""time"" > now())Close, but not good enough.now( ) We're TalkingWith our latest release, TimescaleDB 2.7, we approached things differently, adding an optimization that would allow the evaluation ofnow()expressions during plan-time chunk exclusion.Looking at the root of the problem, the reason whynow()would not be correct is due to prepared statements. If you executenow()but only use that value in a transaction half an hour later, the value does not reflect thecurrent time—now()—anymore.However,it will still hold true for certain expressions even as time goes by.For example,time >= now()will be true at this moment, in 5 minutes and 10 hours. So, when optimizing this, we looked for expressions that held as time passed and used those during plan-time exclusion.The initial implementation of this feature works for intervals of hours, minutes, and seconds (e.g.,now() - ‘1 hour’).As you can see from the EXPLAIN output, chunks are no longer excluded during execution. The exclusion happens earlier, during planning, speeding up the query. Success!Custom Scan (ChunkAppend) on metrics1k  (cost=0.00..1.02 rows=1 width=20)
   Chunks excluded during startup: 0
   ->  Seq Scan on _hyper_3_39453_chunk  (cost=0.00..1.02 rows=1 width=20)
         Filter: ((""time"" > '2022-05-24 12:41:31.266968+02'::timestamp with time zone) AND (""time"" > now()))In the next TimescaleDB version, 2.8, we are removing the initial limitations of thenow()optimization, making it also available in intervals of months and years. This means that you will be able to make the most of this improvement in a wider range of situations, as anytime > now() - Intervalexpression will be usable during plan-time chunk exclusion.Custom Scan (ChunkAppend) on metrics1k  (cost=0.00..1.02 rows=1 width=20)
   Chunks excluded during startup: 0
   ->  Seq Scan on _hyper_3_39453_chunk  (cost=0.00..1.02 rows=1 width=20)
         Filter: (""time"" > now())This code is alreadycommittedin ourGitHub repo, and will be available shortly.How Does It Work?But how did we make this current version happen? The optimization works by rewriting the constraint. For example:time > now() - INTERVAL ‘5 min’turns into((""time"" > (now() - '00:05:00'::interval)) AND (""time"" > '2022-06-10 09:58:04.224996+02'::timestamp with time zone))This means that the constified part of the constraint will be used during plan-time chunk exclusion. And, assuming that time only moves forward, the result will still be correct even in the presence of prepared statements, as the original constraint is ANDed with the constified value.Rewriting the constraint makes the constified value available to plan-time constraint exclusion, leading to massive reductions in planning time, especially in the presence of many chunks.So we know that this translates into faster queries. But how fast?Performance Comparison—now( ) That Is Fast!As shown in our table, the optimization’s performance improvement scales with the total number of chunks in the hypertables. The more data partitions you’re dealing with, the more you’ll notice the speed improvement—up to 401x faster in TimescaleDB 2.7for a total of 20,000 chunks when compared to the previous version.now()that is fast. 🔥The table lists the total execution time of the query (at the beginning of the post) on hypertables with a different number of chunksnow( ) Go Try ItThere are few things more satisfying for a developer than solving a problem for your users, especially a recurring one. Achieving such performance optimization is just the icing on the cake.If you want to experience the lightning-fast performance of PostgreSQLnow()queries for yourself, TimescaleDB 2.7 is available for Timescale and self-managed TimescaleDB.If you are a Timescale user, you will be automatically upgraded to TimescaleDB 2.7. No action is required from your side. You can also create a free Timescale account to geta free 30-day trial(no credit card required).If you are using TimescaleDB in your own instances,check out our docs for instructions on how to upgrade.Once you’re using TimescaleDB, connect with us! You can find us in ourCommunity Slackand theTimescale Community Forum. We’ll be more than happy to answer any question on query performance improvements, TimescaleDB, PostgreSQL, or other time-series issues.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/how-we-fixed-long-running-postgresql-now-queries/
2022-09-29T12:57:19.000Z,What’s New in TimescaleDB 2.8?,"TimescaleDB 2.8 is now available onTimescaleand fordownload. This major release includes the following new features:Thetime_buckethyperfunction now supports bucketing by month, year, and time zone, enabling easier time-based queries and reporting.You can now manage refresh, compression, and retention policies for continuous aggregates all in one easy step.Improved performance of bulkSELECTstatements that fetch high volumes of data from distributed hypertables using theCOPYprotocol.Support forON CONFLICT ON CONSTRAINT UPSERTstatements on hypertables, which enables better compatibility with some GraphQL/ PostgreSQL middlewares.Let’s explore these improvements in more detail.Thetime_buckethyperfunction now supports bucketing by month, year, and time zone“While the newtime_bucketfunctionality is rather simple, its impact is immense and simplifies analytical requests with the need of time zone information.”Chris Engelbert, “Nightmares of Time Zone Downsampling: Why I’m Excited About the Newtime_bucketCapabilities in TimescaleDB”Support for time zones and bucketing your data by monthly or yearly intervals using thetime_buckethyperfunctionwere some of the most requested features, and they’re finally available in TimescaleDB 2.8.Developed with thecommunity's helpover the last couple of months, the new implementation initially lived in TimescaleDB’sexperimental namespaceastime_bucket_ng. The functionality has since been improved through testing and feedback and is now generally available as part of thetime_buckethyperfunction. These capabilities simplify analytical requests with the need for time zone information. By providing atime zoneparameter in thetime_buckethyperfunction, developers can now adjust theoriginaccording to the given time zone. That means that those daily, monthly, or yearly boundaries are also modified automatically. When migrating existing code, all your current queries will work just as they did before the change.Using monthly or yearly buckets and specifying a time zone is simple, as illustrated in the query below:SELECT
   time_bucket('1 month', created, 'Europe/Berlin') AS bucket,
   avg(value)
FROM metrics
GROUP BY 1📚Read more aboutimprovements to thetime_buckethyperfunction.Create and manage multiple policies for continuous aggregates in one stepContinuous aggregateshelp developers query large amounts of time-series datamore quickly, and are one of the hallmark features of TimescaleDB. Prior to TimescaleDB 2.8, you could only add one policy at a time, which could be tedious and complicated. Now with TimescaleDB 2.8’s one-step policy for continuous aggregates, you can add, remove, or modify multiple policies of continuous aggregates with a single command, including refresh, compression, and retention policies. Note that this feature is experimental, so we welcome your feedback on how to improve it for production use.Watch thedemo videoor read ourdocumentationto learn more about one-step policy management for continuous aggregates.Performance improvements for distributed hypertables using theCOPYprotocol for high data volume queriesIn TimescaleDB 2.8, we improved the performance of queries that select a high volume of data from distributed hypertables, such asSELECT *.... For this, we started to use the so-calledCOPYprotocol—the subset of the PostgreSQL native protocol used to efficiently transfer data in bulk. We also made our code more efficient and reduced overhead by streamlining how we work with the internal data structures used to hold and transfer the row data.Hypertables now supportON CONFLICT ON CONSTRAINT UPSERTstatementsWith the TimescaleDB 2.8 release,hypertablesnow support theON CONFLICT ON CONSTRAINTclause, which fixes a long-standing compatibility issue with GraphQL/PostgreSQL middlewares like Hasura, Prisma, and Postgraphile.Try TimescaleDB 2.8If you are using Timescale, upgrades are automatic, and you’ll be upgraded to TimescaleDB 2.8 in your next maintenance window.New to Timescale?Start a free 30-day trial, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/whats-new-in-timescaledb-2-8/
2020-04-24T19:46:40.000Z,"TimescaleDB 1.7: fast continuous aggregates with real-time views, PostgreSQL 12 support, and more Community features","We’ve just finished our latest release🥳.Here’s a quick look at what’s new and how to get started.Fresh from engineering, we’re excited to announce TimescaleDB 1.7, which includes two major updates: PostgreSQL 12 support and Real-time Aggregates. In addition, we’ve made several data management features available in our free Community edition.PG 12 SupportTimescaleDB 1.7 addsPostgreSQL 12support, aligning TimescaleDB with the latest Postgres features.  As always, with TimescaleDB you can build on a proven PostgreSQL ecosystem and leverage Postgres extensions - if it works with Postgres, it works with TimescaleDB.Introducing Real-time AggregatesTimescaleDB 1.7 introduces real-time aggregates, which return real-time results forcontinuous aggregatesover selected time intervals.  Now, queries oncontinuous aggregates(originally introduced in TimescaleDB 1.3) will automatically return the pre-calculated materialized view combined with the latest raw data available at query time, so you always receive up-to-date results.Real-time aggregates close the delta between the last time data is materialized and query time.Real-time aggregates build on continuous aggregatesTimescaleDB continuous aggregates make it really fast to get aggregate answers by precomputing values (such as the min/max/average value over each hour). For example, if you’re collecting raw data every second, querying hourly data over the past week without continuous aggregates means you'd be processing 60 x 60 x 24 x 7 = 604,800 values at query time.  With continuous aggregates, querying hourly data over the past week means reading 24 x 7 = 168 values from the database - clearly a faster query.  But, while continuous aggregates offer huge performance advantages, they don't incorporate the very latest data (i.e., since the last time the asynchronous aggregation job ran inside the database).Now, with real-time aggregates, a single, simple query will combine your pre-computed hourly rollups with the raw data from the last hour, to always give you an up-to-date answer.  Instead of touching 604,800 rows of raw data, the query reads 167 pre-computed rows of hourly data and 3600 rows of raw data available at query time, leading to significant performance improvements.Real-time aggregates is now the default behavior fornewcontinuous aggregates. To enable real-time query behavior on existing continuous aggregates, use this configuration setting:ALTER VIEW continuous_view_name SET (timescaledb.materialized_only=false);You can also disable the default real-time aggregates behavior - seethis sectionof our documentation for more details.Coming down the pipeline: real-time aggregates in-depthNext week, we’ll publish a detailed real-time aggregates blog, showing you different ways to use it and example queries. In the meantime, you can viewthe full release notes and ChangeLog on GitHub– andreach out to us on Slackwith questions at any time.Data management features in (free) Community EditionWe’ve made features that simplify how you manage your data available for free, in our Community Edition, including:Data re-ordering: set policies to reorder data on disk, optimizing for performance, and better manage data that’s queried in an orderdifferentfrom when it’s collected.Data retention:  create, remove, or alter policies to automatically “drop” (delete) older chunks on a defined schedule.Summary and ResourcesTimescaleDB 1.7 real-time aggregates help you perform fast SQL analysis across massive, continuously aggregated datasets, while still getting a real-time view over the latest data.Ready to try it out?If you're an existing TimescaleDB user, follow these upgradeinstructions.If you are brand new to TimescaleDB, get startedhere.For more information on getting started with real-time aggregates, check out ourdocs.And, if you’re new to continuous aggregates - the base technology of real-time aggregates - and want to learn more, check outthe continuous aggregates tutorial.If you have any questions along the way, we are always available via our communitySlack(our engineers, co-founders, and community members are active in all channels).Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescaledb-1-7-fast-continuous-aggregates-with-real-time-views-postgresql-12-support-and-more-community-features/
2021-05-26T12:14:27.000Z,TimescaleDB 2.3: Improving Columnar Compression for Time-Series on PostgreSQL,"TimescaleDB 2.3 makes built-in columnar compression even better by enabling inserts directly into compressed hypertables, as well as automated compression policies on distributed hypertables.Time-series data is relentless. In order to measure everything that matters, you need to ingest thousands, perhaps even millions, of data points per second. The more data you’re able to collect, the more in-depth your analysis is bound to be.  Yet storing and analyzing this data does not come for free, and volume of time-series data can lead to significant storage cost.TimescaleDB addressed this storage challenge through a novel approach to database compression. Rather than just compressing entire database pages using a single standard compression algorithm, TimescaleDB uses multiplebest-in-class lossless compression algorithms(which it chooses automatically based on the type and cardinality of your data), along with a unique method to create hybrid row/columnar storage. This provides massive compression savings and speeds up common queries on older data.We firstintroduced compression in TimescaleDB 1.5, amidst much excitement about how a row-oriented database could support columnar compression. Users were delighted with 92-96% compression rates on their time-series data (likejerezeandVitaliy), leading to both significant storage and cost savings for their databases, as well as query performance improvements. Our approach didn’t just save users storage (and thus money), but it made many queries faster!(These performance improvements arise because high compression rates require less data to be read from disk, and also because TimescaleDB can now read specific compressed columns from disk [rather than entire rows], and special placement optimizations further colocate commonly-requested data.)But TimescaleDB’s columnar compression came with some useability limitations, specifically that compressed chunks wereimmutableonce created:  You couldn’t alter the schema of compressed hypertables, and you couldn’t insert or modify compressed chunks. Earlier this year, inTimescaleDB 2.1, we addressed much of the first limitation by enabling users to add and rename columns in compressed hypertables.We’re excited that today’s release makes it even easier for developers to work with compressed hypertables. TimescaleDB 2.3 includes some key improvements to our native compression, making it suitable for almost every use case. These include:You can now directly INSERT into compressed hypertables(vs decompressing, inserting, and then re-compressing chunks manually).Compression is now fully supported for multi-node TimescaleDB.You can configure automated compression policies for distributed hypertables.Read on for more about these improvements to native compression in TimescaleDB 2.3, including the how and the why.If you’re new to TimescaleDB and want to get started today,create a free accountto get started with a fully-managed TimescaleDB instance (100% free for 30 days).If you are an existing user:Timescale:TimescaleDB 2.3 is now the default for all new services on Timescale, and any of your existing services will be automatically upgraded during your next maintenance window.Managed Service for TimescaleDB:TimescaleDB 2.3 will come in the next few weeks.Self-managed TimescaleDB:Here are upgrade instructionsto upgrade to TimescaleDB 2.3.Join ourSlack communityto share your results, ask questions, get advice, and connect with other developers (our co-founders, engineers, and passionate community members are active on all channels).You can alsovisit our GitHubto learn more (and, as always, ⭐️ are appreciated!) And, if these are the types of challenges you’d like to help solve,we are hiring!Shoutout to all the engineers who worked on these features:Gayathri Ayyappan,Sven Klemm,Markos Fountoulakis, andMats Kindahland the entire team of reviewers and testers!Inserting data into compressed hypertablesTimescaleDB’s native compression works by keeping recent data as standard uncompressed database rows, which allow them to be written into the database at very high rates: 100,000s rows per second on single-node, millions of rows per second on multi-node. Then, after some amount of time, the set of rows are compressed into TimescaleDB’s columnar format according to a compression policy. (More specifically, a TimescaleDBhypertable is partitioned into chunksbased on rows’ timestamps, and once the chunk is older than, say, 3 days old, the entire chunk gets compressed.)Before TimescaleDB 2.3, you couldn’t insert into these chunks once compressed.  In most cases this works fine, as many time-series workloads write only recent data (and the user can then specify a “3 day” or “1 month” threshold for compression based on their needs).But sometimes this isn’t sufficient. You might need to insert older data into those compressed chunks on an irregular basis. For example, if some missing or delayed records are added to your data.  Or you might want to do a one-off bulk import of supplemental data, e.g., you run a building management platform and you want to add the historical records of a specific building into your database, or you are storing tick data for crypto or fintech applications and you are importing historical information about a new symbol.So in prior versions of TimescaleDB, you could only perform these backfills to compressed hypertables by going through several manual steps: you first had to identify which chunks your data fit into, decompress the chunks in question, then perform the INSERT of data into those chunks and finally, re-compress the chunks in question.These steps can be tedious, especially when it involves multiple chunks and the chunks are not contiguous. (We previously offered somehelper functions involving temp tablesfor such large backfills, but the user experience still wasn’t straightforward.)From TimescaleDB 2.3 onward, you can now perform INSERTs directly into hypertables with compressed chunks, much like writing to any standard database table.Here’s how it worksThe ProblemWhen you compress a chunk, the data from the original chunk is stored in an internal chunk in compressed form. This poses a problem, as you cannot directly insert a new row into the chunk, as it is in compressed form.As an analogy, say you have a big array of integers, with integers in the range (1,10000) and you compress them usingdelta encoding, so that they are efficiently encoded and packed contiguously as a single data tuple. Now you want to insert a new integer to this compressed series - say the number 120. How do you add this new value to the already compressed array?  There’s no room for it in the physical representation of the compressed array!This is similar to the problem we have with inserting data into compressed chunks in TimescaleDB. Except that we now have multiple columns that could have compressed data. That's why we initially could not support inserts after a chunk is compressed, and required the intermediate steps of decompressing, performing the insert, and then re-compressing the chunk.How we solved the problemBehind the scenes of what happens when you insert new data into a hypertable containing compressed chunks. Rows with recent timestamps are added to uncompressed chunks as normal, and rows with timestamps belonging to compressed chunks are written to internal chunks in compressed form.When you insert a new row into a chunk that has been compressed, we compress this single row and save it in compressed form in the internal chunk. This is why inserting into a compressed chunk has a very small performance penalty compared to inserts to uncompressed chunks: because we are compressing these new rows in a row-by-row manner as they are written. (Thus, if you are doing a bulk backfill of a large amount of historical data, we still recommendour existing backfill approachfor higher ingest performance.)However, that is not the whole story, as INSERTs into a table also have to respect constraints, triggers, etc. that are defined on that table. We make sure that these work exactly as expected when inserting into a previously compressed chunk.Typically after compression, the compressed chunks compress many rows together for greater storage savings. Basically, if a row has a timestamp, label, and value, then we build arrays with 1000 rows of data, so that the 1000 timestamps are grouped and compressed together using a compression algorithm that works well for timestamps, the corresponding 1000 labels are grouped and compressed using a string compression algorithm, and likewise for the 1000 metrics. It’s a bit more complex than that based on thesegment-byandorder-bysettingsdefined on that hypertable, but that’s the basic idea.But since we now have compressed data that contains only 1 row of tuples, we need to periodically recompress the chunk. This way, we can combine these individually-inserted rows into the existing compressed batch of rows, and not lose the storage savings from compression. But because this processing happens asynchronously and as a batch, it’s much more efficient than doing it on every insert.This re-compression of chunks to merge individual compressed rows is executed automatically (i.e., no user intervention is required) as part of TimescaleDB’s job scheduling framework. Automated compression policies in TimescaleDB run according to a defined schedule, so we designed the re-compression step keeping in mind the fact that the compression policy will run again at some point in the near future.Behind the scenes when a compression policy job runs on a hypertable containing compressed chunks that have been inserted into. When the compression policy runs, the compressed chunk will be re-compressed according to its specified “segment_by” and “order_by” settings, resulting in all rows in the chunk being compressed efficiently.Leveraging Asynchrony and BatchingWith this approach, users just insert data into a compressed hypertable like any other database table for a smooth developer experience, while our approach leverages asynchrony and batching for greater efficiency. Consider the alternate approach for handling inserts: identify the ""compressed row"" in the internal chunk that can accommodate this newly inserted row (i.e., the set of 1000 tuples that got compressed together), decompress the set of tuples in this compressed row, add the new individual row to this set, and then compress the result again. This approach would be extremely inefficient and have very bad performance. But by amortizing the cost of decompression across many individual rows and inserts, and performing it only once in an asynchronous manner, our approach supports both higher insert rates and is much more resource efficient.See our documentationfor a list of current limitations to inserting data into compressed hypertables and more information on using the feature.Enabling compression policies on distributed hypertablesTimescaleDB is now adistributed, multi-node, petabyte-scale relational databasefor time series. To achieve multi-node, TimescaleDB 2.0 – released in February 2021 –  introduces the concept of adistributed hypertable. A distributed hypertable is a hypertable that automatically partitions data into chunks across multiple machines, while still maintaining the illusion (and user experience) of a single continuous table across all time.With TimescaleDB 2.3, you can now use automated compression policies on distributed hypertables. For background, acompression policyenables you to automatically compress data older than a certain period according to a schedule. The policy defines the time period after which data should be compressed (for example, after 7 days, 3 months, 1 year, etc.) according to the compression settings enabled on the distributed hypertable.Bringing automated compression policies to multi-node TimescaleDB enables you to enjoy the consistent experience you’re accustomed to with single-node deployments of TimescaleDB: just set up a compression policy and be done with it.Prior to TimescaleDB 2.3, you could only perform an action of manually compressing a specific chunk, rather than just configure an age-based policy.  This could be quite tedious.  For example, if you wanted to compress all data older than 7 days across your distributed hypertable, you would have to write custom code to run the manual compression commands, setup policies on each of the data nodes, or remind yourself to do it every week for data that aged over the past week.Now with TimescaleDB 2.3, to automatically compress all data older that 7 days in your distributed hypertable, you simply run the following commands:First, to configure compression on the distributed hypertable:alter table conditions set (
  timescaledb.compress, 
  timescaledb.compress_segmentby = 'location',
  timescaledb.compress_orderby = 'time');Second, to create the compression policy to continuously compress chunks whenever their data becomes older than 7 days from the current time:select add_compression_policy('conditions', interval '7d');To enable compression policies on a multi-node instance of TimescaleDB, we create a job on the access node that runs the distributed compression policy. The policy utilizes synchronous compression among the data nodes and updates the metadata in the access nodes. See themulti-node documentationfor more details about access nodes, data nodes, and multi-node TimescaleDB environments. But largely, this happens transparently to users: They just set a policy and can forget about it.SummaryTimescaleDB 2.3 makes native compression even better, enabling users to insert data directly into compressed chunks in hypertables, and to create automated compression policies on distributed hypertables.These new features work together to give you that seamless, great developer experience with TimescaleDB. It’s PostgreSQL with superpowers for time-series data.  And we want to take on the complexity of managing time-series at scale (even trillions of rows of data) so you don’t have to worry about it.We’re always working to improve TimescaleDB.  We also welcome suggestions or discussionson our GitHub.  And if you are interested in what we’re working on next, please see ourpublic release notes and future plans. Feedback is always welcome!Learn more and get startedIf you’re new to TimescaleDB,create a free accountto get started with a fully-managed TimescaleDB instance (100% free for 30 days).If you are an existing user:Timescale:TimescaleDB 2.3 is now the default for all new services on Timescale, and any of your existing services will be automatically upgraded during your next maintenance window.Managed Service for TimescaleDB:Support for TimescaleDB 2.3 will be coming in the next few weeks.Self-managed TimescaleDB:Here are upgrade instructionsto upgrade to TimescaleDB 2.3.Join ourSlack communityto share your results, ask questions, get advice, and connect with other developers (our co-founders, engineers, and passionate community members are active on all channels).You can alsovisit our GitHubto learn more (and, as always, ⭐️ are appreciated!) And if these are the types of challenges you’d like to help solve:we are hiring!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/timescaledb-2-3-improving-columnar-compression-for-time-series-on-postgresql/
2023-01-18T16:23:31.000Z,An Incremental Materialized View on Steroids: How We Made Continuous Aggregates Even Better,"Time-series data is often collected at a much higher granularity than is later required for display or historical storage. Having too much data never sounds like a problem until it becomes one for speed and storage reasons.A materialized viewis commonly used to precalculate data for faster access. So, we roll data up (or downsample it) into lower granularity datasets (from minutes to days, for example). Typically, the process continues through multiple stages of rollups, going from seconds to minutes, days, weeks, months, etc.In the past, these several rollups had to be generated from the actual raw dataset. This meant the raw data had to be around for as long as the largest rollup window, likely increasing your storage needs.With the release of TimescaleDB 2.9, we solved this problem by adding support forhierarchical continuous aggregates.Simply put, continuous aggregates on top of continuous aggregates.Materialized View vs. Continuous AggregateContinuous aggregates,which can be described as incremental and automatically updated materialized views, have been part of TimescaleDB for quite a while now. They are one of the most beloved features, enabling users to pre-aggregate data in the background and making it quickly available when necessary.A common use case for continuous aggregates is dashboards, where data is often displayed at a much lower granularity than it was recorded. Imagine a data point like CPU usage, which is recorded at a second’s granularity. It is unlikely to display it at the same granularity level in Grafana.You’d normally use averages or percentiles over the course of a minute or even lower, such as a five-minute window.SELECT
	time_bucket('5 minutes', ""time"") AS ""time"",
	avg(cpu_usage) AS ""avg_cpu_usage""
FROM cpu_usage_metrics
WHERE ""time"" BETWEEN now() - INTERVAL '5 days' AND now()
  AND ""machine_id"" = 42
GROUP BY 1
ORDER BY 1;While you can query this time window directly from the raw data, depending on the amount of data and the ingress granularity, the query may not satisfy your response time requirements or deliver a “laggy” user experience.Pre-calculating the required granularity helps give an instant feel to the dashboard for an amazing user experience.CREATE MATERIALIZED VIEW cpu_usage_metrics_avg_5min
	WITH (timescaledb.continuous) AS
SELECT
	time_bucket('5 minutes', ""time"") AS ""time"",
	""machine_id"",
	avg(cpu_usage) AS ""avg_cpu_usage""
FROM cpu_usage_metrics
GROUP BY 1, 2
ORDER BY 1;Querying the same result data as above is now as simple as any other query:SELECT
	""time"",
	""avg_cpu_usage""
FROM cpu_usage_metrics_avg_5min
WHERE ""time"" BETWEEN now() - INTERVAL '5 days' AND now()
  AND ""machine_id"" = 42
ORDER BY 1;If you need to work with yet another granularity level, let’s say 15 minutes, just create another continuous aggregate with the necessary rollup window, and you’ll be fine. That is, if the raw data is available for rolling up at the internal refresh window, which can lead to issues when you need to roll up data for a monthly time window. All raw data needs to be available at that point in time.Well, not anymore!Added Speed and Storage Savings With Hierarchical Continuous AggregatesWith TimescaleDB 2.9 or later, you can roll up a continuous aggregate from a previous continuous aggregate. That means theFROMclause can reference another continuous aggregate, which wasn’t allowed before.Returning to the 15-minute example, we can now implement a continuous aggregate using the already pre-aggregated five-minute one. For the sake of correctness (since it uses an average, and those can be tricky when using multi-stage averages), let’s slightly change the five-minute continuous aggregate by adding an intermediatesumandcount.CREATE MATERIALIZED VIEW cpu_usage_metrics_avg_5min
	WITH (timescaledb.continuous) AS
SELECT
	time_bucket('5 minutes', ""time"") AS ""time"",
	""machine_id"",
	avg(cpu_usage)   AS ""avg_cpu_usage"",
	sum(cpu_usage)   AS ""sum_cpu_usage"",
	count(cpu_usage) AS ""count_cpu_usage""
FROM cpu_usage_metrics
GROUP BY 1, 2
ORDER BY 1;With that out of the way, the 15-minute continuous aggregate is as simple as the following:CREATE MATERIALIZED VIEW cpu_usage_metrics_avg_15min
	WITH (timescaledb.continuous) AS
SELECT
	time_bucket('15 minutes', ""time"") AS ""time"",
	""machine_id"",
	sum(sum_cpu_usage) / sum(count_cpu_usage) AS ""avg_cpu_usage""
FROM cpu_usage_metrics_avg_5min
GROUP BY 1, 2
ORDER BY 1;As you can see, we don’t use the average function anymore, but the two additional intermediate values to build the average. For other aggregations, it may be easier or more complex depending on the multi-stage aggregation requirements of the algorithm.Anyhow, we end up with pre-aggregated 15-minute slices per machine, just as if we’d calculated it straight from the raw data. The benefit here is that you can already expire and delete the raw data after the initial five-minute window is calculated, dropping the amount of stored data to one-third. Imagine the storage savings with something like a monthly time window.And that’s not only true for the raw data, but every single continuous aggregate can haveits own retention policy, too. Just make sure the data is further aggregated before it's retired and removed.But there is one additional benefit: speed. It’s much faster to average over three values than 900. While it doesn’t make a massive difference at this level, more complex algorithms will be a lot faster based on the number of data points.Diagram example of hierarchical continuous aggregates functionality for a finance use caseAs a quick side note for the termhierarchical; we called the featurehierarchical continuous aggregatessince you may branch out from one continuous aggregate into many. One example would be a continuous aggregate with one-day time slices, which is then aggregated into multiple continuous aggregates, such as seven days, 14 days, one month, etc. The branching can become arbitrarily complex—the limit is your imagination.Try the New Continuous Aggregates With TimescaleDB 2.9Timescale is happy to releasehierarchicalcontinuous aggregateswith TimescaleDB 2.9. It is one of the most requested and wished-for features, and we love to make our users (you!) happy with the functionality that is actually needed.Like always, this is not the only cool new addition to TimescaleDB 2.9. Other features include time zone support fortime_bucket_gapfill(an extension of the time zone support fortime_bucketin 2.8) or fixed schedule support for background jobs. For a complete list, check out theRelease Notes.For Timescale users, upgrades are automatic, and you’ll be upgraded automatically using the next maintenance window.If you are new toTimescale, start your free 30-day trial now, no credit card required, and get your new database journey started in five minutes.If you’re self-hosting TimescaleDB, follow theupgrade instructionsin our documentation.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/an-incremental-materialized-view-on-steroids-how-we-made-continuous-aggregates-even-better/
2021-08-03T12:10:53.000Z,"Move Fast, but Don’t Break Things: Introducing the Experimental Schema (With New Experimental Features) in TimescaleDB 2.4","At Timescale, we have a track record of constantinnovation, moving quickly to solve painful problems that developers who work with time-series data experience (For example, we recentlyshipped 12 launches in a single month!)But on the other hand, we also valuestability– and so do our users. We cannot change our API every time we release a new database version, nor can we expect TimescaleDB users to constantly make changes to their production code in order to keep their database working, or risk breakages if they don’t. The potential damage caused by “moving fast and breaking things” could alienate the very users who rely on our product: developers who expect stability and reliability from their database.At Timescale, we build critical infrastructure components, and our users expect and rely on stability (see our aptly titled“when boring is awesome” launch announcement). As the makers of TimescaleDB, the leading relational database for time-series data, we recognize that there are nearly three million active TimescaleDB databases running mission-critical time-series workloads across industries. Our prime directives are to safeguard data and ensure our customers’ applications do not need to be rewritten with every release, and to continuously build features big and small to make developers’ lives easier.We made the decision early in the design of TimescaleDB to build on top of PostgreSQL. We believed then, as we do now, that building on theworld’s fastest-growing databasewould have numerous benefits for our customers. Among these benefits is the rock-solid dependability developers have come to expect from a database with 20+ years of production usage behind it. Building on top of PostgreSQL enables us to innovate rapidly with solutions for ingesting and analyzing time-series data while also leveraging (and contributing to) a database with a reputation for stability.Striking a balance between innovation and stability is not a problem unique to TimescaleDB, but it’s perhaps the most important question in our software development process. We strive to achieve the dependability developers expect from PostgreSQL, yet also aim to deliver rapid innovation and progress for customers building solutions in an industry that constantly changes.So, how do we resolve the tension between innovation and stability so that we can continue to ship new functionality quickly while retaining the trust we’ve earned with our customers?Today, to reinforce our commitment to moving fast andnotbreaking things,we are introducinga new experimental schema for TimescaleDBas part of our release of TimescaleDB 2.4.The experimentalschemais where we aim to develop and ship features at an even faster pace than we normally do. The general objective is that these new functions will “graduate” out of the experimental schema when they reach full maturity for normal production usage.TimescaleDB 2.4 introduces the following experimental functionality in the new experimental schema:A function to analyze data in arbitrary time intervals that improves on the populartime_bucketfunction, with support for bucketing using months and years and support for time-zone use incontinuous aggregates. Youasked, and we delivered.Functionality for elasticity and high availability in multi-node TimescaleDB. Specifically, functions to copy or move data (at the chunk level) between data nodes.Read on for more about how we introduced experimental functionality in TimescaleDB, details about new experimental functionality, and examples of when and why to use them.TimescaleDB 2.4 is available today for Timescale and self-managed users and will be available in the coming weeks for Managed Service for TimescaleDB users.If you’re new to TimescaleDB and want to try out our new experimental functionality right away,create a free accountto get started (30-day trial 🔥).Join ourSlack communityto share your results, ask questions, get advice, and connect with 7K+ other developers (our co-founders, engineers, and passionate community members are active on all channels).Visit our GitHubto learn more (and, as always, ⭐️ are appreciated!). And, if these are the types of challenges you’d like to help solve,we are hiring!Shoutout to all the engineers who worked on these features: Mats Kindahl, Erik Nordström, Nikhil Sontakke, Aleksander Alekseev, Dmitry Simonenko, and the entire team of reviewers and testers!We’d like to give a special shoutout to all community members who’veasked for improvementsto thetime_bucketfunction and participated in ourdiscussions on GitHub, both of which informed our choice of experimental functionalities introduced today.Introducing the experimental schema in TimescaleDBExperimental schemas enable us to validate new functionality and get feedback from real users, and enable developers to test the latest and greatest of what TimescaleDB has to offer.Moreover, the schema’s explicit experimental name and the “experimental” component of API functions act as a “buyer beware” label and set expectations, telling users to anticipate breaking changes and to not use these features in production until they’ve “graduated” and are stable. This frees our team to make changes to existing features without breaking compatibility with functionality that users rely on and mature features through iterative improvements until they are ready for wider consumption.The experimental schema is an important milestone for our users and us, demonstrating Timescale’s commitment to innovation, not just in the features and products we release but in the process of how we actually build and deliver our products to delight developers. It also gives us a sandbox where users can test the latest features and more actively shape the development of TimescaleDB.We first released experimental functionality inTimescaleDB hyperfunctions,a series of SQL functions within TimescaleDB that make it easier to manipulate and analyze time-series data in PostgreSQL with fewer lines of code.In addition to production-ready hyperfunctions for calculatingtime-weighted averagesandpercentile approximations, we released several hyperfunctions in public preview for developers to trial through our hyperfunctions experimental schema and offer feedback to improve the features so that they may be released for production usage soon. These experimental hyperfunctions included functions for downsampling, smoothing, approximate count-distinct, working with counters, and working with more advanced forms of averaging.How experimental features workAs noted previously, functionality under the experimental schema isexperimentalin nature and not recommended for production use at this time. You should expect that several of these functions may change their APIs, change their names, change views, or disappear. Additionally, when experimental features “graduate” from the experimental schema, they’re not guaranteed to allow moment-in-time upgrades.Here’s how features graduate from the experimental schema to the main schema (and are thus deemed ready for production use):Step 1: We release Feature X in the experimental schema, tell users about it, and gather user feedback. Users give feedback bysubmitting new issues or commenting on existing issues in GitHub.Step 2: Depending on the amount and severity of issues uncovered during the feedback process, we spend time addressing the feedback. This can take weeks to months, and would be one or two minor versions after the experimental version was released (i.e., if the experimental version was released in v2.4, the production-ready version might be ready to use in v2.5 or v.2.6)Step 3: Once we’ve stopped receiving bug reports, we graduate Feature X from experimental to production-ready and notify users in our release notes. On the other hand, if we continue to receive bug reports about Feature X while it is in the experimental schema, we keep it in the experimental schema until we stop receiving bug reports (and the feature passes our own internal QA) and then graduate it in the next available release.Ultimately, customer feedback drives the pace at which features graduate through each step and into production.time_bucket_ng: An upgrade to time_bucketThe first new experimental feature we are releasing today is one of ourmost requested featuresand tops many of our community members' wishlists: an improved version oftime_bucket, with support for month, year, and time zones. Thanks toJean-Francois Labbéfor getting the ball rolling 3 years ago:AGitHub issuefrom TimescaleDB community member Jean-Francois Labbé made in January 2018 asking for support for monthly intervals intime_bucket.Background on time_bucketFor those new to TimescaleDB,time_bucketis among the most popular TimescaleDB hyperfunctions and is used for bucketing and analyzing data for arbitrary time intervals in SQL. For readers familiar with PostgreSQL, you can think oftime_bucketas a more powerful version of thePostgreSQLdate_truncfunction.time_bucketallows for arbitrary time intervals, rather than the standard day, minute, hour provided bydate_trunc, enabling more flexibility in doing analysis using the time periods which most matter to your use case.For example, you could usetime_bucketto find the average temperature in 15-day buckets for a certain city, using the following simple query:-- time_bucket
-- Average temp per 15 day period
-- for past 6 months, per city
-----------------------------------
SELECT time_bucket('15 days', time) as ""bucket""
   ,city_name, avg(temp_c)
   FROM weather_metrics
   WHERE time > now() - (6* INTERVAL '1 month')
   GROUP BY bucket, city_name
   ORDER BY bucket DESC;What’s currently missing in time_bucketWe love hearing from our customers and community members. Our entire team regularly reads and answers questions in ourpublic TimescaleDB Slack community. Our Eon Bot (named after our mascot, Eon) helps us identify long-requested functionality, inviting members of our community to vote on feature requests.We also opened up ourGitHub discussionsas a community exercise on what we should scope related to thetime_bucketfeature. The most voted solution was:I would like to specify a timezone like ""UTC+3"" for a given continuous aggregate so that any time from a source table will be converted to the given timezone and end up in a right bucket (in terms of when the next day starts, etc). Multiple continuous aggregates can be created, each with its own time zone, e.g “UTC” one and “UTC+3” one. I also would like to use timezones like ""Europe/Berlin"". Unlike simple “UTC+3”, such timezones account for daylight saving time and will automatically change time. This means that there can be days with 23 or 25 hours, which is not the case when specifying a specific time zone like CEST and CET.The Eon Bot and GitHub discussions are two examples of how to incorporate community feedback in our feature development process.As you can see, the most common requests we received aretime_bucketrelated, specifically: support for months/days/years intime_bucket, support for timezones intime_bucket, and being able to use both in continuous aggregates. The original issues are as old as three years old, with more than 65 upvotes on the original comment. Although providing the requested functionality may sound simple, it’s not trivial to implement it.There are several reasons for this, but the most important one is that thetime_bucketfunction and continuous aggregates were originally implemented for buckets that are fixed in size, like 10 seconds, 3 hours, or 5 days. However, months and years are variable in size: months range between 28 and 31 days, years range between 365 and 366 days. Breaking this constraint requires many changes throughout a large part of the TimescaleDB code base, and you have to keep backward compatibility and performance in mind. All in all, this is doable, but it takes time.Time zone support is also a case of variable-sized buckets: If there is adaylight saving timechange in a given timezone, it means there are days that have between 23 and 25 hours.We had to re-engineertime_bucketfor these kinds of variable-sized buckets.The next generation of time_bucketEntertime_bucket_ng(NextGeneration) with implemented support for: Years, Months, Weeks, Days, Hours, Minutes, and Seconds.time_bucket_ngalso includes support for use with continuous aggregates. The new support for years and months enables a host of capabilities for analysis that looks over longer time periods (e.g., business dashboards showing month-over-month growth).To make this a bit more concrete, we’ve included a few examples oftime_bucket_ngin action.Here’s how to usetime_bucket_ngto create bucket data in 3-month intervals:-- for our readers from North America: the date is in YYYY-MM-DD format
SELECT timescaledb_experimental.time_bucket_ng('3 month', date '2021-08-05');
 time_bucket_ng
----------------
 2021-07-01
(1 row)Here’s how to usetime_bucket_ngto bucket data in 1-year intervals:SELECT timescaledb_experimental.time_bucket_ng('1 year', date '2021-08-05');
 time_bucket_ng
----------------
 2021-01-01
(1 row)Here’s how to usetime_bucket_ngwhen creating continuous aggregates. In this case, we track the temperature in Moscow over 7-day intervals:CREATE TABLE conditions(
  day DATE NOT NULL,
  city text NOT NULL,
  temperature INT NOT NULL);

SELECT create_hypertable(
  'conditions', 'day',
  chunk_time_interval => INTERVAL '1 day'
);

INSERT INTO conditions (day, city, temperature) VALUES
  ('2021-06-14', 'Moscow', 26),
  ('2021-06-15', 'Moscow', 22),
  ('2021-06-16', 'Moscow', 24),
  ('2021-06-17', 'Moscow', 24),
  ('2021-06-18', 'Moscow', 27),
  ('2021-06-19', 'Moscow', 28),
  ('2021-06-20', 'Moscow', 30),
  ('2021-06-21', 'Moscow', 31),
  ('2021-06-22', 'Moscow', 34),
  ('2021-06-23', 'Moscow', 34),
  ('2021-06-24', 'Moscow', 34),
  ('2021-06-25', 'Moscow', 32),
  ('2021-06-26', 'Moscow', 32),
  ('2021-06-27', 'Moscow', 31);

CREATE MATERIALIZED VIEW conditions_summary_weekly
WITH (timescaledb.continuous) AS
SELECT city,
   	timescaledb_experimental.time_bucket_ng('7 days', day) AS bucket,
   	MIN(temperature),
   	MAX(temperature)
FROM conditions
GROUP BY city, bucket;

SELECT to_char(bucket, 'YYYY-MM-DD'), city, min, max
FROM conditions_summary_weekly
ORDER BY bucket;

  to_char   |  city  | min | max
------------+--------+-----+-----
 2021-06-12 | Moscow |  22 |  27
 2021-06-19 | Moscow |  28 |  34
 2021-06-26 | Moscow |  31 |  32
(3 rows)(Seeour continuous aggregates docsfor more information about continuous aggregates, and check out ourgetting started guidefor tips on how to use them and other TimescaleDB advanced features.)time_bucket_ngdemonstrates the power of using the experimental schema to solve long-standing community requests without introducing breaking changes. It also gives community members the opportunity to give feedback about the new functionality, so we make sure our implementation solves real users’ problems. (Please let us know the results of your testing and new functionality you’d like on ourGitHub issues page.)In the next iterations oftime_bucket_ng, we will release support for: 1 month in continuous aggregates; N-months and N-Years with continuous aggregates; and time zones.The following chart indicates the differences intime_bucketfunctionality:time_bucket_ngadds support for bucketing by years and months and timezone functionality for use in queries, as well as in continuous aggregates.Introducing improved elasticity and high availability for multi-node TimescaleDBWe introducedmulti-node TimescaleDBin TimescaleDB 2.0, and with it, the ability for users to run petabyte-scale workloads across multiple physical TimescaleDB instances (called data nodes). Multi-node TimescaleDB enables horizontal scaling and rapid ingest of the most demanding time-series workloads and was designed from the start to be highly available and elastically expand as data needs grow.However, an important piece of the puzzle has been missing, namely the ability to move and copy data around the cluster (collectively, all the nodes in your deployment). Let’s look at elasticity and high availability separately to see how copying and moving data helps in those situations.Elastically scaling up and downIt is already possible to horizontally expand your multi-node TimescaleDB database by adding a new data node on self-managed TimescaleDB deployments. However, this doesn’t provide immediate value: a new node will join the cluster without any data and will neither serve data nor alleviate pressure on storage or compute for existing nodes. Thus, you need to wait until enough new data is written to the new data node before it’s used in practice.In TimescaleDB 2.4, we introduce a new experimental function (strictly aPostgreSQL procedure) that frees you to move chunks from existing data nodes to new data nodes:CALL timescaledb_experimental.move_chunk(‘_timescaledb_internal._dist_hyper_1_1_chunk’, ‘data_node_2’, ‘data_node_3’);For example, this makes it possible to add a new data node and move old chunks to the new data node to alleviate storage pressure.Conversely, it is also possible to delete a data node, but, in order to preserve the integrity of the database, it is not possible to do it as long as the node still holds data. By moving chunks off the data node, it is now possible to take a data-serving node out of rotation, either to replace it with a differently configured node or simply to scale down the cluster in order to reduce costs.TimescaleDB already includes amove_chunkfunction that allows users to move chunks across tablespaces (i.e., disks). Our first approach was to extend this existing function to work acrossnodes.However, moving a chunk across two data nodes is a multi-transactional operation, and standard PostgreSQL functions can’t run multiple transactions. Therefore, we had to use aPostgreSQL procedureinstead of a function, which would introduce a breaking change to the currentmove_chunkfunction.With the experimental schema, we can now introduce multi-node scaling much faster for users who’d like to try it out and continue to improve the functionality, without impacting existing interfaces for production use cases. In the future, we might deprecate the existing move function in favor of a unified interface.High availabilityMulti-node TimescaleDB already supports high availability, achieved by usingPostgreSQL’s streaming replicationto add standby(s) to each node in the cluster. However, for a large multi-node cluster, this can be a management-intensive and costly solution where the extra resources aren’t optimally used.As an alternative to the traditional standby setup, multi-node TimescaleDB offers built-in functionality to replicate data at the chunk level. This allows each data node to work both as a primary for some chunks and backup for others. If a data node fails, its chunks already exist on other nodes that can take over the responsibility of serving them.However, after a node failure, there will be some chunks that no longer have the desired number of replicas to sustain further failures, i.e., those chunks are under-replicated. In order to get back to a fully replicated multi-node cluster, an under-replicated chunk needs to be copied to a data node that doesn’t already have a copy. The target node can either be an existing data node or a newly added one.The experimental schema offers a copy function, which is similar to the move function:CALL timescaledb_experimental.copy_chunk(‘_timescaledb_internal._dist_hyper_1_1_chunk’, ‘data_node_2’, ‘data_node_3’);Failures when copying and moving chunksA copy or move operation for a chunk might fail or be interrupted, and in that case, the incomplete operation has to be rolled back.However, as mentioned above, a move or copy happens over multiple transactions so the operation cannot be completely rolled back by PostgreSQL. Instead, TimescaleDB tracks each step of a move or copy, so failed operations can be undone (or resumed).In the experimental schema we have introduced a function to clean up a failed copy or move:CALL timescaledb_experimental.cleanup_copy_chunk_operation('ts_copy_1_31');The function needs to be passed the operation ID that will be logged in case a copy or move operation fails.Provide feedback and help drive the projectWe invite – and encourage – all community members to try the experimental schema in a test-safe environment and to tell us about your experience.What should we improve?Were things easy to understand, or was something difficult to set up or tricky to use?How did you expect a feature would work and did it meet (or exceed!) your expectations?We want to hear it all: let us know about the glitches you encountered, anything that surprised and delighted you, and everything in between.To share feedback,create a GitHub issue(using theexperimental-schemalabel), describe what you found, and tell us the steps - or share a code snippet - to recreate it.We’re committed to developing these and future experimental features – and to giving the community a chance to provide early feedback and influence the direction of TimescaleDB’s development. We’ll travel faster with your input and we truly appreciate your contribution(s) (thank you in advance 🙌!).Get started todayIf you’re new to Timescale and want to get started with our new experimental functionality today,you can self-host to get started. If you want to try our engineeredPostgreSQL cloud platform, try our 30-day free trial(no credit card required).If you are an existing user:Timescale:TimescaleDB 2.4 is now the default for all new services on Timescale, and any of your existing services will be automatically upgraded during your next scheduled maintenance.Managed Service for TimescaleDB:TimescaleDB 2.4 will come in the next few weeks.Self-managed TimescaleDB:Here are upgrade instructionsto upgrade to TimescaleDB 2.4.As a reminder, you can join ourSlack communityto share your results, ask questions, get advice, and connect with other developers (our co-founders, engineers, and passionate community members are active on all channels).You can alsovisit our GitHubto learn more (and, as always, ⭐️  are appreciated!) And, if these are the types of challenges you’d like to help solve,we are hiring!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/move-fast-but-dont-break-things-introducing-the-experimental-schema-with-new-experimental-features-in-timescaledb-2-4/
2021-10-07T13:02:06.000Z,Grow Worry-Free: Storage Autoscaling on Timescale,"❗As Timescale evolves its vision, becoming a simpler, faster, and more cost-effective PostgreSQL (or PostgreSQL++ as we call it), in June 2023, we moved from the autoscaling storage model described in this blog post to a usage-based storage model. Yes, that's right: with Timescale, you are only charged for the data you store in your services.Learn more about our database storage model by reading our announcement.Time-series datais everywhere, and it drives decision-making in every industry. Time-series data collectively represents how a system, process, or behavior changes over time. Understanding these changes helps us to solve complex problems across numerous industries, includingobservability,financial services,Internet of Things, and evenprofessional American football. Analyzing this data across the time dimension enables developers to understand what is happening right now, how that is changing, and why that is changing.And time-series is also relentless, often at high rates and volumes. To help developers keep up, we built Timescale: a database cloud for relational and time-series workloads, built on PostgreSQL, and architected aroundour vision of a modern cloud service:  easy, scalable, familiar, and flexible.We recently announcedlarger disk plansof up to 16TB per node on Timescale, enabling developers to effectively store and analyze 200+ TB of data (thanks to TimescaleDB’s hybrid row-column native compression, with compression rates of 94-97%).Today, to continue with this theme of delivering a worry-free platform for time series that scales with your data needs,we’re announcing the release of storage autoscaling onTimescale.What is storage autoscaling?Storage autoscaling helps you keep your database operations running smoothly and avoid the hassle of dealing with full disks by automatically increasing your database storage with user-defined limits to keep costs under control (and avoid unexpected bills). Storage autoscaling (and manual disk resizing) happens fully online without service downtime.Storage autoscaling ison by default. You now create a database service on Timescale with a single click. Your service starts small, but the platform continuously monitors your storage consumption and automatically increases your capacity when needed. From 10GB up to 16TB (or 200+TB of effective storage). Easy to use and highly cost-effective.But Timescale oud also empowers developers with greater control. The platform allows developers to easily set configurable limits for auto-scaling (or turn it on or off on a per-service basis). When enabled with limits (which can be modified at any time), your storage will automatically scale up to your specified limit.  And then otherfull disk protection mechanismswill kick in if and when you hit your limits.With storage autoscaling enabled, your Timescale service will increase its storage once it reaches 85% of the current storage plan until it reaches its storage autoscaling limit.While managed database services save you the hassle of administering your own database, storage autoscaling helps you save even more time by avoiding dealing with operational blockages caused by full disks, and at the same time ensuring that your users and customers experience uninterrupted service. Furthermore, storage autoscaling gives you full control over your future costs, with transparent pricing and user-defined limits to ensure that you never scale unexpectedly beyond your budget.In addition to automatically scaling up storage as required, Timescale also includes full disk protection mechanisms, which puts the database into read-only mode once the autoscaling limit is reached and the database is full, ensuring that the database doesn’t crash from new data attempting to be inserted into it (read more here about Timescale's storage safety mechanisms.)  It’s another example of how Timescale is focused on providing a worry-free platform for developers.Learn more about storage autoscalingRead on for more about what storage autoscaling is, the problem it solves, and why and how to use storage autoscaling in Timescale.For existing Timescale users, storage autoscaling is enabled by default on all new services created, just visit the Autoscaling options under the Operations tab in your console.If you’re new to Timescale,create a free accountto get started with a fully-managed Timescale service (100% free for 30 days, no credit card required).Once you are using TimescaleDB, pleasejoin the Timescale communityand ask any questions you may have about time-series data, databases, and more.Special thanks toIvan Tolstosheyevand the entire Timescale team behind the development of this feature 🙏.Full Disks Are a PainWe’ve all experienced it. You’re trying to take that latest puppy video or photo to share in your group text, but your storage is full. You’re left trying to figure out which caches to clear or other pictures to delete.Nobody likes a full disk, including your database and the operating system it uses. You try to insert more data into your database, and the “write-ahead-log” (WAL) it uses to ensure all data is reliably and atomically written has no place to write its log. Try to add an index, and there’s no place on disk to store the index pages. And, even if you don’t directly write any new data to the database, things are happening (or, perhaps more accurately, not happening) in the background. Temp files can’t be written. File system blocks can't be allocated. Unexpected things go wrong.As adatabase cloudthat prioritizes ease and scale, Timescale provides built-in safety mechanisms to deal with these issues on your TimescaleDB services.Timescale will alert you whenever you approach storage limitations on your account, put your service in a read-only state so that your data is not lost, and give you an opportunity to configure your service so that you can resume collecting everything that matters.Many of these mechanisms are transparent and “just work” (as one example, Timescale employs system balloon files for an additional layer of “defense in depth” to full disks), and you can read more about the existing measure in our blog post announcingautomated disk management.But today, we wanted to share a bit more about our new capabilities for storage autoscaling and why we built them.There is plenty of friction involved in triaging a database with a full disk (even with protections in place to ensure a database doesn’t crash, like putting it into read-only mode). It requires manual intervention from developers in order to provision more storage or perhaps to move, delete or compress data to save space. This intervention at worst takes away from engineering time that could be used for feature development, or at a minimum is just plain inconvenient.Moreover, in the case you choose to provision more storage, it’s difficult to know exactly how much storage you’ll need with most DBaaS or serverless solutions. If you choose a disk that’s too small, you run the risk of having to manually intervene again in the near future, but if you choose a disk size that’s too big, you end up paying for more capacity than you actually need as you wait for your disk to fill up.Storage autoscaling on Timescale was designed to solve these problems by removing the pain of dealing with full disks and minimizing the time that developers spend worrying about their database.Enter Storage AutoscalingHow it worksWith storage autoscaling on Timescale, you can start with as little storage as you need but set an autoscaling limit to give you the peace of mind that you’ll be able to scale up smoothly to storage levels you may need in the future.With storage autoscaling enabled, when a service hits 85% of the current storage capacity, the storage plan will autoscale to the next available plan. (For example, if you’re on the 25GB storage plan, you’ll be autoscaled to the 50GB. And if you’re on the 9TB plan, you’ll be autoscaled to the 10TB plan.)This means that you’ll never pay for more than the storage plan your service is currently using and only pay for larger storage plans when you autoscale to that tier. You can start at 10GB and automatically scale up to 16TB as your workload increases over time with no manual intervention. Moreover,  storage autoscaling (and manual disk resizing) happens fully online, without service downtime.Autoscaling limitsYou set an autoscaling limit, which is the maximum disk capacity that your service’s storage will reach with autoscaling enabled. This gives you full control over your future costs and helps keep your operations running smoothly with predictable, transparent pricing.The Autoscaling option in the Timescale console shows you your storage autoscaling limit and the next storage plan it will autoscale to.This saves you from potentially nasty billing surprises that many users have experienced on serverless data platforms, where their services automatically scale opaquely without any insight into limits or why their costs increased.What can you do if your limit is reached?Once your autoscaling limit has been reached, your disks enter “read-only” mode, meaning data can no longer be inserted. From here, you’ll be notified to take action. You have three options: first, you can reduce disk usage by compressing data and altering your native compression settings, second, you can enable data retention policies or manually drop chunks, or third, you can update your autoscaling limit to a higher tier. You can alter your autoscaling configuration anytime under your service’s operations tab.Get Started TodayFor existing Timescale users, storage autoscaling is enabled by default on all new services created, just visit the Autoscaling options under the Operations tab in your console.If you’re new to Timescale,create a free accountto get started with a fully-managed Timescale service (100% free for 30 days, no credit card required).Once you are using TimescaleDB, pleasejoin the Timescale communityand ask any questions you may have about time-series data, databases, and more.Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/grow-worry-free-storage-autoscaling-on-timescale-cloud/
2021-10-28T13:15:09.000Z,Massive Scale for Time-Series Workloads: Introducing Continuous Aggregates for Distributed Hypertables in TimescaleDB 2.5,"Announcing TimescaleDB 2.5Time-series datais everywhere, representing how systems, processes, behaviors, and values change over time. By analyzing data across the time dimension, developers can understand what is happening right now, how that is changing, and why that is changing. Time-series data helps developers solve problems in observability, product and user analytics, fintech and blockchain applications, and the Internet of Things, to name but a few industries.And time-series data is also relentless. Because of the sheer volume and rate of information, time-series data can be complex to store, query, and analyze.This challenge (and need!) is why we builtTimescaleDB, a multi-node, petabyte-scale, completely free relational database for time-series. Multi-node TimescaleDB is a distributed system built with the most demanding time-series workloads in mind, scaling to over 10 million metrics ingester per second, storing petabytes of data and processing queries even faster thanks to better parallelization, yet remains built on the rock-solid foundation and maturity of PostgreSQL.Today, there are nearly three million active TimescaleDB databases running mission-critical time-series workloads across industries. This includes a growing number of multi-node deployments, from startups to major enterprises, such as a Fortune 100 tech company that has deployed two production clusters of multi-node TimescaleDB, each with more than 20 servers, to ingest more than a billion rows of telemetry events per day per cluster.TimescaleDB is built for time-series data, and how we designed its scale-out architecture is no exception. While most distributed database systems use one-dimensional sharding to partition data (e.g., by device, stock ticker, or some other application identifier), multi-node TimescaleDB leverages two-dimensional, fine-grained partitioning for greater flexibility, particularly around time-oriented data lifecycle management. This architecture enables not only distributed operation, but also flexible elasticity and data rebalancing, node heterogeneity, data retention policies, efficient data tiering, and more. See ourannouncement blogfor more on the design of multi-node TimescaleDB.One of our goals for multi-node TimescaleDB is to provide a developer experience that’s very similar to that of single-node TimescaleDB, so that eventually, a developer can start with a small single-node deployment, grow over time, and scale-out automatically when needed. Towards this end, recent releases have added functionality to multi-node TimescaleDB to close this gap, such as distributed compression policies, triggers, and generated columns for distributed operations, distributed restore points, additional distributed query push-down optimizations, and more.Continuous aggregates for distributed hypertablesWith today’s release of TimescaleDB 2.5, we are excited to release a highly-requested new capability:continuous aggregates support for distributed hypertables.Continuous aggregates pre-compute results, so that queries on the pre-computed results can be orders of magnitude faster. But this isn’t just a simple key-value cache. With TimescaleDB continuous aggregates, users can query these pre-computed results with the full power of SQL and a powerful query engine, enabling a user to JOIN results against other relational tables, re-aggregate or rebucket the results (e.g., grouping minutely data into 10 minute buckets, or aggregating across many devices), and more.While continuous aggregates are similar to materialized views in PostgreSQL, unlike materialized views, they are incrementally updated with a built-in refresh policy that makes the aggregates stay up-to-date. TimescaleDB’s continuous aggregates also properly handle late or out-of-order data (with minimal overhead), and support the real-time aggregation of pre-computed and raw data. More on this below.Today’s release of continuous aggregates on distributed hypertables retains the same user experience and API that developers already know with hypertables on single-node TimescaleDB. Now, developers can compute continuous aggregates in a multi-node TimescaleDB cluster, massively speeding up workloads that need to process large volumes of data at the petabyte scale in a distributed environment.It’s just a few commands to create a distributed hypertable and continuous aggregate:-- Create a distributed hypertable partitioned on time and device
CREATE TABLE conditions (
   time TIMESTAMPTZ NOT NULL,
   device_id TEXT NOT NULL,
   temperature DOUBLE PRECISION NULL
);
SELECT create_distributed_hypertable('conditions', 'time', 'device_id');

-- Insert some data into the distributed hypertable
INSERT INTO conditions VALUES
   (now() - interval '15m', 'A001', 70.0),
   (now(), 'A001', 90.1),
   (now(), 'B015', 68.5),
   (now(), 'C183', 69.4),
   (now(), 'E602', 73.8);
   
-- Create a continuous aggregate on the distributed hypertable
CREATE MATERIALIZED VIEW conditions_hourly 
   (hour, device_id, mintemp, maxtemp)
WITH (timescaledb.continuous) AS
   SELECT time_bucket('1hour', time), device_id,
      min(temperature), max(temperature)
   FROM conditions
   GROUP BY time_bucket('1hour', time), device_id;

-- Query the continuous aggregate on the distributed hypertable
SELECT * FROM conditions_hourly
   WHERE hour > now() - interval '7day' AND maxtemp > 90;In addition to continuous aggregates for distributed hypertables, this release also adds PostgreSQL 14 support, new time bucketing functionality for expanded timezone management, and a number of other bug fixes.See the release notes for a complete list of all the changes!Want to learn more about how to use continuous aggregates in your distributed hypertables, together with other TimescaleDB features like data retention policies? Watch this demo byMiranda Auhl, Developer Advocate here at Timescale:PostgreSQL 14 supportWe are very proud to announce that TimescaleDB 2.5 includes support for PostgreSQL 14, which was releasedone month ago. As supporters of the Postgres community, we want our users and customers to benefit from the latest and greatest that Postgres has to offer when using TimescaleDB.See the main improvements introduced on this new version of Postgres.We made the decisionearly in the design of TimescaleDBto build on top of PostgreSQL. We believed then, as we do now, that building on theworld’s fastest-growing databasehas numerous benefits for our users and customers. Among these benefits is the rock-solid dependability that developers have come to expect from a database with 20+ years of production usage behind it, so that all the goodness and innovation happening in PostgreSQL itself also benefits TimescaleDB users. Building on top of PostgreSQL enables us to innovate rapidly with solutions for ingesting and analyzing time-series data while also leveraging (and contributing to) a database with a reputation for stability.And in fact, while we were working on releasing support for PG14, our own development and testing processesuncovered a serious memory corruption bug in PG14for which we proposed a patch to the upstream.Timezone support in next-gen time bucketingAdditionally, TimescaleDB 2.5 introduces new timezone support totime_bucket_ng, enabling developers to bucket and analyze their data according to a specific timezone - which is particularly useful for applications spanning data or devices worldwide.Thistime_bucket_ngfunction serves as theexperimental“nextgeneration” version oftime_bucket, but rewritten to introduce support for non-fixed time intervals (specifically, months and years, which have a variable duration based on which month or leap year). Currently,time_bucketonly supports fixed time intervals (e.g., seconds, minutes, hours, days, weeks), which led to a (surprisingly complex) rewrite intime_bucket_ngfor more flexible time bucketing, including for use in continuous aggregates. We plan to graduate this function out of the experimental schema in future releases.Getting TimescaleDB 2.5TimescaleDB 2.5 is available immediately for Timescale and self-managed deployments, and will be available in the coming weeks on Managed Service for TimescaleDB.If you’re new to TimescaleDB,create a free accountto get started with a fully-managed TimescaleDB instance (with a free 30-day trial, no credit card required 🔥).You can alsovisit our GitHubto learn more (and, as always, ⭐️ are appreciated!), or join ourSlack communityto share your results, ask questions, get advice, and connect with 7K+ other TimescaleDB enthusiasts. And, if you are interested in getting alotmore involved,we are hiring worldwide!Interested in learning more about this new functionality? Read on for a refresher on multi-node TimescaleDB and continuous aggregates and how we built support for continuous aggregates on distributed hypertables.Shoutout to all the engineers who worked on features in TimescaleDB 2.5: Markos Fountoulakis, Mats Kindahl, Aleksander Alekseev, Alexsander Kuzmenkov, Sven Klemm, Gayathri Ayyappan, Erik Nordström, Fabrizio Mello, Dmitry Simonenko, Nikhil Sontakke, Duncan Moore, and the entire team of reviewers and testers!We’d also like to give a special shoutout to all community members who’ve asked for improvements to multi-node TimescaleDB,time_bucket_ng, and PostgreSQL 14 support, all of which informed our choice of functionality introduced today.A refresher on multi-node TimescaleDBMulti-node TimescaleDBenables users to run petabyte-scale workloads across multiple physical TimescaleDB instances, allowing users to scale beyond the limits of a single TimescaleDB (or PostgreSQL) instance. To do this, we introduced the concept of adistributed hypertable.A regularhypertable, one of TimescaleDB’s original innovations, is a virtual table in TimescaleDB that automatically partitions data into many sub-tables (“chunks”) on a single machine, continuously creating new ones as necessary, where each chunk includes data belonging to a specific range of timestamps. Yet, it provides the illusion of a single continuous table across all time (the “hypertable”).Adistributed hypertableis a hypertable that is spread across multiple machines, while still maintaining the illusion (and user experience) of a single continuous table across all time. When using the recommended time-and-space partitioning, each chunk in a distributed hypertable is defined by both a time interval and some subset of the keys belonging to the additional partition key. These chunks are automatically created across the distributed hypertable’s data nodes.The multi-node TimescaleDB architecture consists of an access node (abbreviated as AN), which stores metadata for the distributed hypertable and performs query planning across the cluster, and a set of data nodes (abbreviated as DNs), which store individual (or replicated) chunks of the distributed hypertable, and execute queries against those chunks locally. TimescaleDB remains a single piece of software for operational simplicity; any database can become an access node when a user executes theadd_data_nodecommand, passing in the hostnames of other nodes.A distributed hypertable covering one access node (AN) and three data nodes (DN1, DN2, DN3). A distributed hypertable is partitioned into chunks by both “time” and “space” dimensions, which are spread dynamically across available data nodes.Once multi-node TimescaleDB is set up, creating a distributed hypertable is as simple as creating a regular hypertable, as also shown above:-- Create a distributed hypertable partitioned on time and device
CREATE TABLE conditions (
   time TIMESTAMPTZ NOT NULL,
   device_id TEXT NOT NULL,
   temperature DOUBLE PRECISION NULL
);
SELECT create_distributed_hypertable('conditions', 'time', 'device_id');Whenever data is inserted, the access node routes each data tuple to the proper chunk(s) on one or more of its data nodes (in this example, based on timestamp and device id). And as time progresses, the access node will automatically create new chunks on data nodes (corresponding to new time ranges and space partitions) as data continues to be ingested. This ensures that data is spread across the data nodes evenly.Today's multi-node TimescaleDB supports several capabilities which make it suited for large-volume time-series workloads, including efficient query pushdown,automated compression policies,improved data rebalancing for elasticity andhigh availability, distributed object management (e.g., keeping roles, UDFs, and other objects consistent across nodes), and more, all while striving to provide the same experience and functionality as single-node TimescaleDB.A refresher on continuous aggregates (and why they’re powerful)Users are often interested inaggregatescomputed over data, such as the average sensor reading per device per day, the OHLC (open high low close) values per stock per interval, the maximum CPU utilization per 5 minutes, the number of distinct visitors on each web page per day, and so forth.PostgreSQL has powerful support for computing various forms of aggregates, and TimescaleDB’shyperfunctionsadd even more, including various approximate statistics and approximation/sketching algorithms. But computing the aggregates for large tables can be expensive, simply because you have to read and process all the data belonging to the desired interval.To avoid the constant recomputation each time you want to read the aggregated data, many relational databases, including PostgreSQL, support materialized views. The use of materialized views avoid constant recomputation at query time, but come with three limitations for time-series data:The materialized view is not automatically refreshed, so it is necessary to do a regular refresh of the materialized view as new data comes in.When refreshing the materialized view, all the data is read again to compute the aggregate, even if only a small portion of the data is changed.This recomputation necessitates retaining all the underlying raw data in order to perform the refresh of the materialized view.TimescaleDB’scontinuous aggregatesare designed to overcome these three limitations.Continuous aggregates look and behave like materialized views, but they are incrementally updated with a built-in refresh policy that makes the aggregates stay up-to-date as new data is added. Further, they can efficiently handle late or out-of-order data: if data that is inserted, updated, or deleted actually has an older timestamp, the aggregates corresponding to that older period of time are correspondingly updated. But, through the use of clever internal “invalidation records”, the refresh procedure is careful to only refresh data in the materialized view that actually needs to be updated, which avoids recomputing data that did not change. This smart refresh procedure massively speeds up the incremental maintenance of materialized views, and ensures that the aggregate data is up-to-date.Continuous aggregates similarly are amenable to data retention policies that are often applied in time-series workloads, such that aggregates about old data can be retained in the database, even after the corresponding raw data itself has been dropped.When defining a continuous aggregate policy, you define a refresh window. The continuous aggregate will be updated inside this window only, and only the small portions of data that have been changed will be updated, according to internal invalidation records that track changes to old data. Data outside the refresh window won't be refreshed, enabling data retention policies to delete old raw data while still preserving the aggregates.TimescaleDB’s continuous aggregates also supportreal-time aggregation. With real-time aggregation, when querying a continuous aggregate, the query engine will transparently combine (i) results that are already pre-computed in the materialized view with (ii) results computed at query time over the newest raw data in underlying hypertable. (You can turn this off if desired, but the majority of developers want this behavior by default).Real-time aggregation gives you the best of both worlds: the performance of continuous aggregates and the most up-to-date data for real-time queries, without the performance degradation of querying and aggregating all raw data from scratch. This makes real-time aggregation an ideal fit for many real-time monitoring, dashboarding, and analysis use-cases.Continuous aggregates on distributed hypertablesIn TimescaleDB 2.5, these features from continuous aggregates have been extended to distributed hypertables, and use an identicalSQL CREATEcommand as with regular hypertables.But in a multi-node architecture, the continuous aggregate is stored to optimize query performance. In particular, the underlying incrementally materialized view that’s part of a continuous aggregate is stored on the access node, while the raw data continues to reside across the data nodes. Data is asynchronously materialized from data nodes to the access node via the continuous aggregates refresh policy.Queries on continuous aggregates over distributed hypertables achieve high performance. Rather than querying the raw data distributed on the various data nodes and aggregating it at query time, many or all of the results from a continuous aggregate query are already pre-computed and present on the access node. This turns a distributed query across multiple nodes into a local one, while of course also benefiting from results that have been pre-computed as part of the background materialization process, further minimizing the computation needed at query time.Architecture diagram showing how continuous aggregates are stored on a multi-node TimescaleDB cluster. The continuous aggregate is stored on the access node (AN), while the raw data is distributed between the data nodes (DNs). The raw data is materialized in the background by the continuous aggregate refresh policy.Developers familiar with TimescaleDB might wonder what this design means for real-time aggregates. While the pre-computed results in the continuous aggregate might already reside on the AN, the latest raw data (to be aggregated at query time) still resides across the data nodes. But continuous aggregates, even though now on distributed hypertables, continues to support this transparently: the query engine will push down aggregation to the needed DNs for the latest raw data, if a user’s query to a continuous aggregate includes a recent time range that has yet to be materialized.For example, if the continuous aggregate typically has materialized data up until the last 30 minutes, then queries that only touch data older than 30 minutes only need to involve the aggregate data already materialized on the access node. If a query involves newer data, however – say, “give me the max temperature per hour for the past six hours” – then the query engine combines this pre-aggregated (on the AN) with real-time aggregated data from the last 30 minutes (computed across the DNs).This diagram illustrates what happens when you query a continuous aggregate on a distributed hypertable. If the new query involves new data which arrived since the last refresh of the continuous aggregate, the query engine combines the pre-aggregated data on the access node (AN) with this new not yet materialized data from the data nodes (in this case, DN 3) at query time. Note that this query will be considerably more efficient than querying raw data altogether, as the majority of results are pre-computed already on the access node - and only the latest data not yet materialized is fetched at query time from data node 3.Real-time aggregation remains enabled by default, although it can be disabled anytime (including for individual sessions). Without real-time aggregation, a query against a continuous aggregate will be performed fully against data already pre-computed on the access node.Supporting continuous aggregates across distributed hypertables did introduce some new technical challenges. Because the distributed hypertable is spread across multiple data nodes, each data node needs to maintain its own invalidation log to reflect changes to the distributed hypertable’s chunks that reside locally. Then, the refresh process needs to ensure that the proper regions are recomputed, even though these invalidation logs are distributed, and that materialization results from different data nodes are properly serialized (and avoid deadlocks) on the access node. One key step to processing these invalidation logs and refreshing the continuous aggregate involves executing all operations as distributed transactions within the multi-node cluster (using TimescaleDB’s two-phase commit protocol). This way, each stage remains atomic, as it is when the process takes place on single-node TimescaleDB.In future releases, we plan to add functionality to distribute continuous aggregates themselves across data nodes (e.g., where the materialization hypertable underlying the continuous aggregate is itself distributed across the data nodes). This would enable much larger continuous aggregates than can be supported today (e.g., those that cannot fit on the access node).Get started todayTimescaleDB 2.5 is available today for Timescale andself-managed deploymentsand will be available in the coming weeks on Managed Service for TimescaleDB.If you’re new to Timescale,create a free accountto get started with a fully-managed TimescaleDB instance (with a free 30-day trial, no credit card required 🔥).Join ourSlack communityto share your results, ask questions, get advice, and connect with 7K+ other developers (our co-founders, engineers, and passionate community members are active on all channels).Visit our GitHubto learn more (and, as always, ⭐️ are appreciated!). And, if these are the types of challenges you’d like to help solve,we are hiring!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/massive-scale-for-time-series-workloads-introducing-continuous-aggregates-for-distributed-hypertables-in-timescaledb-2-5/
2022-11-16T14:00:00.000Z,"Expanding the Boundaries of PostgreSQL: Announcing a Bottomless, Consumption-Based Object Storage Layer Built on Amazon S3","We are excited to announce the initial launch, in private beta, of our new consumption-based, low-cost object storage layer in Timescale. This new capability expands the boundaries of traditional databases, allowing you to transparently tier your data across disk and Amazon S3 while accessing it as if it all lived in one single continuous PostgreSQL table. This means that you can now store an infinite amount of data in Timescale, paying only for what you store. Bottomless cloud storage for time series, events, and analytics is just one piece of our vision to empower you with exceptional data infrastructure so that you can build the next wave of computing.We started Timescale five years ago with a mission: to help developers build the next wave of computing through applications that leverage time-series and real-time analytical data.This mission led us to build TimescaleDB, a time-series database that gives PostgreSQL the performance boost it needs to handle relentless streams of time-series data at scale.On top of scalability and performance, another key concern for developers managing data at scale iscost efficiency. Time-series data is often collected at high frequency or across long time horizons. This scale is often a fundamental part of applications: it’s storing metrics about all IoT devices in a fleet, all the events in a gaming application, or tick data about many financial instruments. But this data adds up over time, often leading to difficult trade-offs about which data to store and for how long.To address this problem, we’ve developed several database features at Timescale aimed at making it easier for developers to manage their time-series data—likenative columnar compression,downsampling,data retention policies, anduser-defined actions. And indeed, these offer massive savings in practice. By compressing data by 95 percent, Timescale ends up much more cost-effective than vanilla storage options like Amazon RDS for PostgreSQL.Today we’re excited to announce how we’re extending this vision to a cloud-native future and building Timescale to supercharge PostgreSQL for time series, events, and analytics at greater scale and lower cost.Timescale now offers consumption-based, low-cost object storage built on Amazon S3.This new storage layer gives you, the developer, more tools to build applications that scale more efficiently while reducing costs. Leveraging a cost-efficient storage layer like Amazon S3 removes the need to pre-allocate—and pay for—an upper bound of your storage. When you tier data on Timescale, you will only pay for what you actually store while retaining the flexibility tokeep a limitless amount of data,and without being charged extra per query.This consumption-based pricing is not only transparent butan order of magnitude cheaperthan our standard disk-based storage. And what’s more, you can access this affordable object storage layer seamlessly from your Timescale database, meaning no need to create a custom pipeline to archive and reload data. All you’ll need is a single SQL command to automatically tier data based on its age, as suited to your application’s needs:# Create a tiering policy for data older than two weeks
SELECT add_tiering_policy ('metrics', INTERVAL '2 weeks');But why stop at cost efficiency? At Timescale, we strive to create aseamless developer experiencefor every feature we release. That means doing the heavy technical lifting under the covers while you continue interacting with your data in the simplest way possible.When applied to cost-saving object storage in Timescale, this means that even when data is tiered, you can continue to query it from within the database via standard SQL, just like you do in TimescaleDB and PostgreSQL. Predicates, filters, JOINs, CTEs, windowing, andhyperfunctionsall work! Reading data directly from tiered object storage only adds a few tens of milliseconds of latency—and this cost goes away for larger scans.We’ve natively architected Timescale databases to support tables (hypertables) that can transparently stretch across multiple storage layers. The object store is thus an integral part of your cloud database rather than just an archive.Here’s an example of theEXPLAINplan for a query that fetches data from disk and object storage (notice theForeign Scan):EXPLAIN SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;


QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
                -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text)The ability to keep your regular and tiered data both accessible via SQL helps you avoid the silos and application-level patchwork that come from operating a separate data warehouse or data lake. It will also help you escape the operational work and extra costs of integrating yet another tool into your data architecture.Starting today, tiering your data to object storageis available for testing in private beta for all Timescale users.Sign up for Timescaleand navigate to the Operations screen, pictured below, to request access. Timescale is free for 30 days, no credit card required.You can request access to our private beta via the Timescale UIBut, this is just the beginning.We plan to further improve object storage in Timescale to not only serve as bottomless, cost-efficient storage but also as ashared storage layerthat makes it dramatically easier for developers to share data across their entire fleet of databases.This is a huge step forward in our vision to build a data infrastructure that extends beyond the boundaries of a traditional database: combining the flexibility of a serverless platform with all the performance, stability, and transparency of PostgreSQL that developers know and love. Not just a managed database in the cloud, but a true “database cloud” to help developers build the next wave of computing.So yes, we are just getting started.✨ A huge “thank you” to the team of Timescale engineers that made this feature possible, with special mention to Gayathri Ayyappan, Sam Gichohi, Vineetha Kamath, and Ildar Musin.To learn more about Timescale’s new data tiering functionality, how it redefines traditional cloud databases, and how it can help you build scalable applications more cost-efficiently, keep reading.Bottomless Storage for PostgreSQLHaving native access to a cloud-native object store means you can now store an infinite amount of data, paying only for what you store. You no longer have to manually archive data to Amazon S3 to save on storage costs, nor import this data into a data warehouse or other tools for historical data analysis. Timescale’s new data tiering feature moves the data transparently to the object store and keeps it available to the Timescale database at all times.To enable this new functionality in PostgreSQL, we built new database internal capabilities and external subsystems. Datachunks(segments of data related by time) that comprise a tiered hypertable now stretch across standard storage and object storage. We also optimized our data format for each layer: block storage starts in uncompressed row-based format and can be converted to Timescale’snative compressed columnar format.On top of that, all object storage is in a compressed columnar format well-suited for Amazon S3 (more specifically,Apache Parquet). This allows developers more options to take advantage of the best data storage type during different stages of their data life cycle.Once a data tiering policy is enabled, chunks stored in our native internal database format are asynchronously migrated into Parquet format and stored in S3 based on their age (although they remain fully accessible throughout the tiering process). A single SQL query will pull data from the disk storage, object storage, or both as needed, but we implemented various query optimizations to limit what needs to be read from S3 to resolve the query.We perform “chunk exclusion” to avoid processing chunks falling outside the query’s time window. Further, the database doesn’t need to read the entire object from S3, even for selected chunks, as it stores various metadata to build a “map” of row groups and columnar offsets within the object. The result? It minimizes the amount of data to be processed, even within a single S3 object that has to be fetched to answer queries properly.Cost-Effective ScalabilityTimescale’s new object storage layer doesn’t just give PostgreSQL bottomless storage but also gives you, the developer, more tools to build applications that scale cost-efficiently.By leveraging Amazon S3, you no longer have to pre-allocate (and pay for) an upper bound of your storage.While Timescale already offers disk auto-scaling, your allocation is still “bumped up” between predefined levels: from 50 GB to 75 GB to 100 GB, from 5 TB to 6 TB to 7 TB, etc. Our new object storage layer scales effortlessly with your data, and you only pay for what you store.These storage savings can be meaningful: an order of magnitude cheaper than employing standard disk-based storage like EBS.So why isn’t this standard for all databases? We build solutions focused on analytical and time-series data. We are doing these transparent optimizations at the larger chunk level rather than the much smaller database page level. This way, we can effectively make the most of S3, which is optimized—for both price and performance—for larger objects. This same approach wouldn’t be practical when employing traditional page-based strategies for database storage.It’s Still Just PostgreSQL, But BetterAs we say,Timescale supercharges PostgreSQL for time series and analytics. But it’s always been important for us to maintain the full PostgreSQL experience, which developers trust and love. This is why we built TimescaleDB as an “extension” of PostgreSQL (although that “extension” has certainly gotten bigger and bigger over the years!).In our books, a smooth developer experience means that developers can continue interacting with all their data as if it’s a standard table—we do the heavy technical lifting under the covers. It should be invisible, and the more our improvements fade into the background, the better.Developers don’t realize that hypertables are actually heavily partitioned data tables—with thousands of such partitions—they just treat them like standard tables. Developers don’t see Timescale’sreal-time aggregationscombining incrementally pre-aggregated data with the latest raw table data to provide them with up-to-date results every time. They are meant to “just work.”We titled our 2017 launch post“When Boring is Awesome: Building a Scalable Time-Series Database on PostgreSQL.”We still strive to make Timescale seem “boring” to developers—simple, fast, scalable, reliable, and cost-effective so that developers can focus their precious time and minds on building applications.This focus on the developer experience similarly motivated our design of transparent data tiering. When data is tiered, you can continue to query tiered data from within the database via standard SQL—predicates and filters, JOINs, CTEs, windowing, andhyperfunctionsall just work.And what’s more, your SQL query will pull relevant data from wherever it is located: disk storage, object storage, or both, as needed, without you having to specify anything in the query.Here’s what it would look like working with relational and time-series data in Timescale, including tiered data. This example shows the use of sensor data, as you might have for IoT, building management, manufacturing, or the like. After creating tables and hypertables forsites,devices, andmetrics, respectively, you use a single commandadd_tiering_policyto set up a policy that automatically tiers data older than two weeks to low-cost object storage.# Create relational metadata tables, including GPS coordinates 
# and FK constraints that place devices at specific sites
CREATE TABLE sites (id integer primary key, name text, location geography(point)); 
CREATE TABLE devices (id integer primary key, site_id integer references sites (id), description text);

# Create a Timescale hypertable
CREATE TABLE metrics (ts timestamp, device_id integer, value float);
SELECT create_hypertable ('metrics', 'ts');

# Create a tiering policy for data older than two weeks
SELECT add_tiering_policy ('metrics', INTERVAL '2 weeks');Now, after you’ve inserted data into your metrics hypertable (and other relational data devices and site information into your relational tables), you can query it as usual. Reading tiered data only adds an extra latency of around tens of milliseconds, and this latency cost may even go away for larger scans.The following SQL query returns the maximum value recorded per device, per day, for a specific site—a fairly standard monitoring use case. Rather than showing the data results (which will just look normal!), we’ll show the output ofEXPLAIN.This command allows developers to see the actual query plan that will be executed by the database, which in this case includes aForeign Scanwhen the database is accessing data from S3. (With our demo data, three chunks remain in standard storage, while five chunks are tiered onto S3.)EXPLAIN SELECT time_bucket('1 day', ts) as day,
        max(value) as max_reading, 
        device_id  	
    FROM metrics 
    JOIN devices ON metrics.device_id = devices.id 
    JOIN sites ON devices.site_id = sites.id
WHERE sites.name = 'DC-1b'
GROUP BY day, device_id
ORDER BY day;

QUERY PLAN                                                      
----------------------------------------------------------
GroupAggregate
    Group Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
    -> Sort
        Sort Key: (time_bucket('1 day'::interval, _hyper_5666_706386_chunk.ts)), _hyper_5666_706386_chunk.device_id
        -> Hash Join
            Hash Cond: (_hyper_5666_706386_chunk.device_id = devices.id)
            -> Append
                -> Seq Scan on _hyper_5666_706386_chunk
                -> Seq Scan on _hyper_5666_706387_chunk
                -> Seq Scan on _hyper_5666_706388_chunk
                -> Foreign Scan on osm_chunk_3334
            -> Hash
                -> Hash Join
                    Hash Cond: (devices.site_id = sites.id)
                    -> Seq Scan on devices
                    -> Hash
                        -> Seq Scan on sites
                           Filter: (name = 'DC-1b'::text)Replace Your Siloed Database and Data WarehouseTimescale’s new data tiering functionality expands the boundaries of a traditional cloud database to incorporate features typically attributed to data warehouses or data lakes.The ability to tier data to Amazon S3 within Timescale saves you the manual work of building and integrating up a custom system or operating a separate data store (e.g., Snowflake) for your archival of historical data. Instead of setting up, maintaining, and operating a separate system alongside your production database (and a separate ETL process), you can simply work with a Timescale hypertable that serves your entire data lifecycle, where data is distributed across different storage layers.As we’ve illustrated, you can query regular and tiered data seamlessly from this table and also JOIN it to the rest of your tables, avoiding silos without adding more complexity to your data stack. This not only simplifies operations but also billing: unlike regular data warehousing systems (which typically charge per query, making it very difficult to forecast the final cost), in Timescale you’ll pay only for what you store, keeping your pricing transparent at all times.Request Access to Data Tiering TodayIf you’re already using Timescale,you can test data tiering today by requesting access to our private beta. We welcome your feedback to improve the product and better serve the needs of developers.To start testing data tiering in Timescale today:Sign up to Timescale.The first 30 days are completely free (no credit card required).Log in to the Timescale UI. In your Service screen, navigate to Operations > Data Tiering. Click on the “Request Access” button, and we’ll be in touch soon with the next steps.Bottomless cloud storage for time series, events, and analytics is just one piece of our vision to empower you, the developer, with exceptional data infrastructure so that you can build the next wave of computing.We plan to further improve object storage in Timescale to not only serve as bottomless, cost-efficient storage but also as ashared storage layerthat makes it dramatically easier for developers to share data across their entire fleet of databases.If that sounds interesting to you, please request access to the private beta and let us know!We’re just getting started!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/expanding-the-boundaries-of-postgresql-announcing-a-bottomless-consumption-based-object-storage-layer-built-on-amazon-s3/
2022-03-29T13:00:00.000Z,Introducing One-Click Database Forking in Timescale,"[Note: This blog post was originally published in December 2021 and updated in March 2022 to reflect new functionality released as part of #AlwaysBeLaunching Cloud Week🐯 ☁️.]Announcing improved support for database forking in Timescale. Create a fork of your database, with the same or different resource configurations, in just a few clicks. Forks allow you to conveniently create testing and staging environments, safely test major upgrades, downsize your service, give access to production data while fully isolating the production database, and much more. 🔥We’re excited to continue #AlwaysBeLaunchingMOAR Edition—a week full of exciting new features for Timescale, bringing you MOAR features that makeTimescaleeven MOAR worry-free, scalable, and flexible!Today, we’re releasing improved one-click database forking in Timescale, with new functionality that allows you to easily spin up forks— identical copies of your database—but now withdifferentresource configurations (CPU, memory, or disk) than the primary database.One-click database forking on Timescale gives development teams the ability to perform a variety of important tasks in less time and with more flexibility: what previously took a considerable amount of manual work can now be done in one or two clicks.Database forking is useful for dev, test, and staging environments, including testing for performance regression, application changes, or safe database upgrades.  It similarly allows you to easily evaluate the impact of schema, index, or configuration changes outside your production database. That is, forks help grow your confidence with any in-place production changes.In addition to only taking one or two clicks to create, database forking on Timescale is cost-effective and significantly reduces the risk of testing on your own production database. Forked databases are billed at an hourly rate, so you only pay for the time your forked instance was active. This means that you can safely test the impact of PostgreSQL version upgrades, changes in application code, a new database INDEX, and more for just pennies per hour.Moreover, with the new ability to upsize or downsize your database fork, you have greater control over resource consumption. For example, if turning on compression reduces your Timescale storage consumption by 90 %, you can now fork into a service with much smaller configured storage, leading to lower costs. Or because forking is a seamless way to grant access to your production data (but not to the production database) to data science or business intelligence teams without needing a separate ETL job or data pipeline, you can easily create forks with more or less resources depending on the needs of their projects.To fork a service in Timescale, follow these simple steps:Select the database service you’d like to fork.Navigate to the “Operations” tab, and click the “Fork service” button.Choose your configuration. To create a fork with the same configuration as the parent service, click on “Create fork.” To fork with a different configuration as the parent service, click instead on “Advanced options,” which will allow you to select your forked service's compute and disk size.That’s it!You can create forks with the same or different resource configurations in TimescaleAfter the time of forking, your forked service will be a completely independent database.  While it inherits the settings of its parent, any subsequent changes in the forked service won’t be reflected in the parent database (and vice versa). In the Timescale UI, you will see which database service was forked from.All forked databases maintain their identity to their parent service, making it easy to remember which database the service was originally forked from.Database forking is available to all users of Timescale. If you’re new to Timescale,you can create a free account to get started(100 % free for 30 days and no credit card required).Have questions or feedback? Join our community! You can chat with us in ourCommunity Slack, and for in-depth technical questions, you can use theTimescale Forum. Feel free to ask us anything!Keep reading for more information on how forking works under the hood, get some ideas for how to simplify your development workflows, and get insights on our roadmap for forking in Timescale.A huge thank you to all the engineers and designers who worked on this feature.The Many Uses of Database ForkingThe ability to easily fork your database comes in handy for multiple scenarios. Here are some common ones:Testing.If your team needs a common image of your database for running correctness or performance regression tests, we recommend leveraging forks in the following way.  (1) Create a ""golden image"" of your database by creating schemas, loading data, etc., but then (2) pause your service so that you'll only pay for storage rather than compute costs, and you can prevent unwanted modifications. (3) Now create a fork of your paused database service, and (4) run any testing against the running service.  (5) Finally, once your team is finished, delete the fork. You can repeat this process as regularly as you need in your testing environment. And if you ever want to tweak your base service, just resume it against and treat it like a normal database.Moreover, you can test the impact of different workloads on your database and find the optimal resource configuration to deal with them, helping you reduce uncertainty for seasonal events. For example, you might anticipate a high load event on your database but aren't sure exactly what resources you'd need. You can create forks with different resource configurations and stress test against them to see which configuration performs best.Safer database upgrades.When doing a major upgrade in a production database (e.g., upgrading from PostgreSQL 12 to 13), we recommend forking your service first. This way, you can perform the upgrades on the forked service first, ensuring that there are no issues related to this change. Once you’re sure the upgrade was successful, you can be confident that everything will work well when running it on your production service.Downsize your service.Forks also allow you to easily downsize your service to eliminate unnecessary costs. In certain situations, you may find yourself paying for a significant amount of unused storage; perhaps your data size has been reduced considerably after enabling or optimizingTimescaleDB compression, or perhaps you’re seeing less traffic than originally predicted. If that happens to be the case, you can conveniently downsize your service using forks, i.e., (1) you can create a fork of your original service, assigning it less disk size and/or compute, and (2) connect your application to the fork, which now acts as your primary database. Once your old and more expensive service is no longer active, you can delete it. And if your data volume increases, you can scale up your service again by simply navigating to the “Resources” tab and selecting a larger storage plan.Create and refresh staging environments.An important aspect of a good testing procedure is having a staging environment with production data so you can test the quality of your new features in real-world conditions. Through database forking, you can spin up an exact copy of your production data without affecting the actual production service. Also, as your production data changes over time, it is good practice to refresh your staging service, as the conditions may change—to have an easy procedure for forking makes this task painless.Provide access to production data (but not to the production database).Many times, teams of data scientists or business intelligence analysts might want access to production data to query and analyze. Database forking enables you to provide access to production data, without having to provide access to the database itself. This is especially useful for cases where implementing access control via PostgreSQL might be too complicated. Simply fork the database and provide teams access to a copy of that data and a dedicated connection string while your production database continues to operate unaffected.You can also give the forked database a different set of resources than the original database, depending on the needs of other teams it's being shared with and the duration of their projects. For example, you could create a fork with more CPU/RAM resources if your data science or BI team is doing a short-term project with heavy analytical or OLAP queries—this helps those teams move quickly as they won't have to wait as long for queries to execute while leaving your production database and its operations unaffected.Create a snapshot of your data.Keeping database snapshots can be very useful for auditing and reporting, and also for doing potential analysis or forensics after carrying out an important change in your service configuration.Forking: Under the HoodEvery service running in Timescale has a backup that we regularly test. This backup is more than a snapshot of the data directory: powered by the continuous archiving feature of PostgreSQL, it contains all database changes at a given point in time. It can be used to restore a database even when the original volume containing the data directory is unavailable.To fork a service in Timescale, instead of restoring the backup of the parent service in place, it is restored to a new instance (the fork) that becomes a clone of the original one.See our blog post oncontinuous backup/restore validation on Timescalefor more details.What’s Next?At Timescale, we like to move fast(without breaking things) so this is only the beginning for database forking in Timescale. In the near future, we will release the following functionality:Forking to a different region than the parent service.Forking from an arbitrary point in time (PIT)—other than the latest—so you can fork to older states of your database.A programmatic API to automate forking.One-click forking for multi-node services.Timescale fully supports multi-node deployments, but for now, forking is only available for single-node databases.Stay tuned, and let us know if there’s more forking functionality you’d like to see.And may the forks be with you!Get StartedDatabase forking is available to all users of Timescale.Check out our documentationfor more information on forking.If you are new to Timescale:Create an accountto get free access to Timescale for 30 days, with no credit card required—and start forking today!Join the Timescale Community Slackand ask us any questions about time-series data, TimescaleDB, PostgreSQL, and more. Join us: we are 8,000+ and counting!Read ourvision for Timescale: a database cloud for relational and time-series workloads, built on PostgreSQL and architected around our vision of a modern cloud service: easy, scalable, familiar, and flexible.Check out our Getting Started documentation. These articles walk you through the basics: creating your first instance, accessing your database, loading your data, and so on. Get familiar with key TimescaleDB concepts likehypertables and chunks,compression, orcontinuous aggregates. Understanding these key features will allow you to use Timescale to its full potential. ✨And, for those who share our mission of serving developers worldwide 🌏 and want to join our fully remote, global team,we are hiring broadly across many roles!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/introducing-one-click-database-forking-in-timescale-cloud/
2022-03-30T13:15:09.000Z,Using pg_stat_statements to Optimize Queries,"pg_stat_statementsallows you to quickly identify problematic or slow Postgres queries, providing instant visibility into your database performance. Today, we're announcing that we've enabledpg_stat_statementsby default in all Timescale services. This is part of our #AlwaysBeLaunching Cloud Week with MOAR features!🐯☁️PostgreSQL is one of the fastest-growing databases in terms of usage and community size, being backed by many dedicated developers and supported by a broad ecosystem of tooling, connectors, libraries, and visualization applications. PostgreSQL is also extensible: usingPostgreSQL extensions, users can add extra functionality to PostgreSQL’s core.  Indeed, TimescaleDB itself ispackaged as a PostgreSQL extension, which also plays nicely with the broad set of other PostgreSQL extensions, as we’ll see today.Today, we’re excited to share thatpg_stat_statements, one of the most popular and widely used PostgreSQL extensions, is now enabled by default in all Timescale services. If you’re new to Timescale,start a free trial(100 % free for 30 days, no credit card required).What is pg_stat_statements?pg_stat_statementsis a PostgreSQL extension that records information about your running queries. Identifying performance bottlenecks in your database can often feel like a cat-and-mouse game. Quickly written queries, index changes, or complicated ORM query generators can (and often do) negatively impact your database and application performance.How to use pg_stat_statementsAs we will show you in this post,pg_stat_statementsis an invaluable tool to help you identify which queries are performing slowly and poorly and why. For example, you can querypg_stat_statementsto know how many times a query has been called, the query execution time, the hit cache ratio for a query (how much data was available in memory vs. on disk to satisfy your query),  and other helpful statistics such as the standard deviation of a query execution time.Keep reading to learn how to querypg_stat_statementsto identify PostgreSQL slow queries and other performance bottlenecks in your Timescale database.A huge thank you to Lukas Bernert, Monae Payne, and Charis Lam for taking care of all things pg_stat_statements in Timescale.How to Querypg_stat_statementsin TimescaleQuerying statistics data for your Timescale database from thepg_stat_statementsview is straightforward once you're connected to the database.SELECT * FROM pg_stat_statements;

userid|dbid |queryid             |query                         
------+-----+--------------------+------------------------------
 16422|16434| 8157083652167883764|SELECT pg_size_pretty(total_by
    10|13445|                    |<insufficient privilege>      
 16422|16434|-5803236267637064108|SELECT game, author_handle, gu
 16422|16434|-8694415320949103613|SELECT c.oid,c.*,d.description
    10|16434|                    |<insufficient privilege>      
    10|13445|                    |<insufficient privilege>   
 ...  |...  |...                 |...Queries that thetsdbadminuser does not have access to will hide query text and identifierThe view returns many columns of data (more than 30!), but if you look at the results above, one value immediately sticks out:<insufficient privilege>.pg_stat_statementscollects data on all databases and users, which presents a security challenge if any user is allowed to query performance data. Therefore, although any user can query data from the views, only superusers and those specifically granted thepg_read_all_statspermission can see all user-level details, including thequeryidandquerytext.This includes thetsdbadminuser, which is created by default for all Timescale services. Although this user owns the database and has the most privileges, it is not a superuser account and cannot see the details of all other queries within the service cluster.Therefore, it's best to filterpg_stat_statementsdata byuseridfor any queries you want to perform.-- current_user will provide the rolname of the authenticated user
SELECT * FROM pg_stat_statements pss
	JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user;


userid|dbid |queryid             |query                         
------+-----+--------------------+------------------------------
 16422|16434| 8157083652167883764|SELECT pg_size_pretty(total_by
 16422|16434|-5803236267637064108|SELECT game, author_handle, gu
 16422|16434|-8694415320949103613|SELECT c.oid,c.*,d.description
 ...  |...  |...                 |...Queries for only thetsdbadminuser, showing all details and statisticsWhen you add the filter, only data that you have access to is displayed. If you have created additional accounts in your service for specific applications, you could also filter to those accounts.To make the rest of our example queries easier to work with, we recommend that you use this base query with a common table expression (CTE). This query form will return the same data but make the rest of the query a little easier to write.-- current_user will provide the rolname of the authenticated user
WITH statements AS (
SELECT * FROM pg_stat_statements pss
		JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user
)
SELECT * FROM statements;

userid|dbid |queryid             |query                         
------+-----+--------------------+------------------------------
 16422|16434| 8157083652167883764|SELECT pg_size_pretty(total_by
 16422|16434|-5803236267637064108|SELECT game, author_handle, gu
 16422|16434|-8694415320949103613|SELECT c.oid,c.*,d.description
 ...  |...  |...                 |...Query that shows the same results as before, but this time with the base query in a CTE for more concise queries laterNow that we know how to query only the data we have access to, let's review a few of the columns that will be the most useful for spotting potential problems with your queries.calls: the number of times this query has been called.total_exec_time: the total time spent executing the query, in milliseconds.rows: the total number of rows retrieved by this query.shared_blks_hit: the number of blocks already cached when read for the query.shared_blks_read: the number of blocks that had to be read from the disk to satisfy all calls for this query form.Three quick reminders about the data columns above:All values are cumulative since the last time the service was started, or a superuser manually resets the values.All values are for the same query form after parameterizing the query and based on the resulting hashedqueryid.The current configuration for Timescale services does not track query planning statistics because of the small added overhead. We may allow this through user configuration in the future.Using these columns of data, let's look at a few common queries that can help you narrow in on the problematic queries.Long-Running PostgreSQL QueriesOne of the quickest ways to find slow Postgres queries that merit your attention is to look at each query’s average total time. This is not a time-weighted average since the data is cumulative, but it still helps frame a relevant context for where to start.Adjust thecallsvalue to fit your specific application needs. Querying for higher (or lower) total number of calls can help you identify queries that aren't run often but are very expensive or queries that are run much more often than you expect and take longer to run than they should.-- query the 10 longest running queries with more than 500 calls
WITH statements AS (
SELECT * FROM pg_stat_statements pss
		JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user
)
SELECT calls, 
	mean_exec_time, 
	query
FROM statements
WHERE calls > 500
AND shared_blks_hit > 0
ORDER BY mean_exec_time DESC
LIMIT 10;


calls|mean_exec_time |total_exec_time | query
-----+---------------+----------------+-----------
 2094|        346.93 |      726479.51 | SELECT time FROM nft_sales ORDER BY time ASC LIMIT $1 |
 3993|         5.728 |       22873.52 | CREATE TEMPORARY TABLE temp_table ... |
 3141|          4.79 |       15051.06 | SELECT name, setting FROM pg_settings WHERE ... |
60725|          3.64 |      221240.88 | CREATE TEMPORARY TABLE temp_table ... |   
  801|          1.33 |        1070.61 | SELECT pp.oid, pp.* FROM pg_catalog.pg_proc p  ...|
 ... |...            |...                 |Queries that take the most time, on average, to executeThis sample database we're using for these queries is based on theNFT starter kit, which allows you to ingest data on a schedule from the OpenSea API and query NFT sales data. As part of the normal process, you can see that aTEMPORARY TABLEis created to ingest new data and update existing records as part of a lightweight extract-transform-load process.That query has been called 60,725 times since this service started and has taken around 4.5 minutes of total execution time to create the table. By contrast, the first query shown takes the longest, on average, to execute—around 350 milliseconds each time. It retrieves the oldest timestamp in thenft_salestable and has used more than 12 minutes of execution time since the server was started.From a work perspective, finding a way to improve the performance of the first query will have a more significant impact on the overall server workload.Hit Cache RatioLike nearly everything in computing, databases tend to perform best when data can be queried in memory rather than going to external disk storage. If PostgreSQL has to retrieve data from storage to satisfy a query, it will typically be slower than if all of the needed data was already loaded into the reserved memory space of PostgreSQL. We can measure how often a query has to do this through a value known as Hit Cache Ratio.Hit Cache Ratio is a measurement of how often the data needed to satisfy a query was available in memory. A higher percentage means that the data was already available and it didn't have to be read from disk, while a lower value can be an indication that there is memory pressure on the server and isn't able to keep up with the current workload.If PostgreSQL has to constantly read data from disk to satisfy the same query, it means that other operations and data are taking precedence and ""pushing"" the data your query needs back out to disk each time.This is a common scenario for time-series workloads because newer data is written to memory first, and if there isn't enough free buffer space, data that is used less will be evicted. If your application queries a lot of historical data, older hypertable chunks might not be loaded into memory and ready to quickly serve the query.A good place to start is with queries that run often and have a Hit Cache Ratio of less than 98 %. Do these queries tend to pull data from long periods of time? If so, that could be an indication that there's not enough RAM to efficiently store this data long enough before it is evicted for newer data.Depending on the application query pattern, you could improve Hit Cache Ratio by increasing server resources, consider index tuning to reduce table storage, or useTimescaleDB compressionon older chunks that are queried regularly.-- query the 10 longest running queries
WITH statements AS (
SELECT * FROM pg_stat_statements pss
		JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user
)
SELECT calls, 
	shared_blks_hit,
	shared_blks_read,
	shared_blks_hit/(shared_blks_hit+shared_blks_read)::NUMERIC*100 hit_cache_ratio,
	query
FROM statements
WHERE calls > 500
AND shared_blks_hit > 0
ORDER BY calls DESC, hit_cache_ratio ASC
LIMIT 10;


calls | shared_blks_hit | shared_blks_read | hit_cache_ratio |query
------+-----------------+------------------+-----------------+--------------
  118|            441126|                 0|           100.00| SELECT bucket, slug, volume AS ""volume (count)"", volume_eth...
  261|          62006272|             22678|            99.96| SELECT slug FROM streamlit_collections_daily cagg...¶        I
 2094|         107188031|           7148105|            93.75| SELECT time FROM nft_sales ORDER BY time ASC LIMIT $1...      
  152|          41733229|                 1|            99.99| SELECT slug FROM streamlit_collections_daily cagg...¶        I
  154|          36846841|             32338|            99.91| SELECT a.img_url, a.name, MAX(s.total_price) AS price, time...

 ... |...               |...               | ...             | ...The query that shows the Hit Cache Ratio of each query, including the number of buffers that were ready from disk or memory to satisfy the queryThis sample database isn't very active, so the overall query counts are not very high compared to what a traditional application would probably show. In our example data above, a query called more than 500 times is a ""frequently used query.""We can see above that one of the most expensive queries also happens to have the lowest Hit Cache Ratio of 93.75 %. This means that roughly 6 % of the time, PostgreSQL has to retrieve data from disk to satisfy the query. While that might not seem like a lot, your most frequently called queries should have a ratio of 99 % or more in most cases.If you look closely, notice that this is the same query that stood out in our first example that showed how to find long-running queries. It's quickly becoming apparent that we can probably tune this query in some way to perform better. As it stands now, it's the slowest query per call, and it consistently has to read some data from disk rather than from memory.Queries With High Standard DeviationFor a final example, let's look at another way to judge which queries often have the greatest opportunity for improvement using the standard deviation of a query execution time.Finding the slowest queries is a good place to start. However, as discussed in the blog postWhat Time-Weighted Averages Are and Why You Should Care, averages are only part of the story. Althoughpg_stat_statementsdoesn't provide a method for tracking time-weighted averages, it does track the standard deviation of all calls and execution time.How can this be helpful?Standard deviation is a method of assessing how widely the time each query execution takes compared to the overall mean. If the standard deviation value is small, then queries all take a similar amount of time to execute. If the standard deviation value is large, this indicates that the execution time of the query varies significantly from request to request.Determining how good or bad the standard deviation is for a particular query requires more data than just the mean and standard deviation values. To make the most sense of these numbers, we at least need to add the minimum and maximum execution times to the query. By doing this, we can start to form a mental model for the overall span execution times that the query takes.In the example result below, we're only showing the data for one query to make it easier to read, the sameORDER BY time LIMIT 1query we've seen in our previous example output.-- query the 10 longest running queries
WITH statements AS (
SELECT * FROM pg_stat_statements pss
		JOIN pg_roles pr ON (userid=oid)
WHERE rolname = current_user
)
SELECT calls, 
	min_exec_time,
	max_exec_time, 
	mean_exec_time,
	stddev_exec_time,
	(stddev_exec_time/mean_exec_time) AS coeff_of_variance,
	query
FROM statements
WHERE calls > 500
AND shared_blks_hit > 0
ORDER BY mean_exec_time DESC
LIMIT 10;


Name              |Value                                                |
------------------+-----------------------------------------------------+
calls             |2094                                                 |
min_exec_time     |0.060303                                             |
max_exec_time     |1468.401726                                          |
mean_exec_time    |346.9338636657108                                    |
stddev_exec_time  |212.3896857655582                                    |
coeff_of_variance |0.612190702635494                                    |
query             |SELECT time FROM nft_sales ORDER BY time ASC LIMIT $1|Queries showing the min, max, mean, and standard deviation of each queryIn this case, we can extrapolate a few things from these statistics:For our application, this query is called frequently (remember, more than 500 calls is a lot for this sample database).If we look at the full range of execution time in conjunction with the mean, we see that the mean is not centered. This could imply that there are execution time outliers or that the data is skewed. Both are good reasons to investigate this query’s execution times further.Additionally, if we look at the coefficient of variation column, which is the ratio between the standard deviation and the mean (also called the coefficient of variation), we get 0.612 which is fairly high. In general, if this ratio is above 0.3, then the variation of your data is quite large. Since we find the data is quite varied, it seems to imply that instead of a few outliers skewing the mean, there are a number of execution times taking longer than they should. This provides further confirmation that the execution time for this query should be investigated further.When I examine the output of these three queries together, this specificORDER BY time LIMIT 1query seems to stick out. It's slower per call than most other queries, it often requires the database to retrieve data from disk, and the execution times seem to vary dramatically over time.As long as I understood where this query was used and how the application could be impacted, I would certainly put this ""first point"" query on my list of things to improve.Speed Up Your PostgreSQL QueriesThepg_stat_statementsextension is an invaluable monitoring tool, especially when you understand how statistical data can be used in the database and application context.For example, an expensive query called a few times a day or month might not be worth the effort to tune right now. Instead, a moderately slow query called hundreds of times an hour (or more) will probably better use your query tuning effort.If you want to learn how to store metrics snapshots regularly and move from static, cumulative information to time-series data for more efficient database monitoring, check out the blog postPoint-in-Time PostgreSQL Database and Query Monitoring With pg_stat_statements.pg_stat_statementsis automatically enabled in all Timescale services. If you’re not a user yet,you can try out Timescale for free(no credit card required) to get access to a modern cloud-native database platform withTimescaleDB's top performance, one-clickdatabase replication,forking, andVPC peering.One last thing: to cap off this #AlwaysBeLaunching Cloud Week, we hosted our secondTimescale Community Day.Check out our talks (and demos!)about all things time-series data and databases!Ingest and query in milliseconds, even at terabyte scale.Try Timescale for free",https://www.timescale.com//blog/using-pg-stat-statements-to-optimize-queries/
