The index on a date column makes sense when you are going to search the table for a given date(s), e.g.:
select * from test
where the_date = '2016-01-01';
-- or
select * from test
where the_date between '2016-01-01' and '2016-01-31';
-- etc
In these queries there is no matter whether the sort order of primary key and the date column are the same or not. Hence rewriting the data to the new table will be useless. Just create an index.
However, if you are going to use the index only in ORDER BY:
select * from test
order by the_date;
then a primary key integer index may be significantly (2-4 times) faster then an index on a date column.
Answer from klin on Stack OverflowThe index on a date column makes sense when you are going to search the table for a given date(s), e.g.:
select * from test
where the_date = '2016-01-01';
-- or
select * from test
where the_date between '2016-01-01' and '2016-01-31';
-- etc
In these queries there is no matter whether the sort order of primary key and the date column are the same or not. Hence rewriting the data to the new table will be useless. Just create an index.
However, if you are going to use the index only in ORDER BY:
select * from test
order by the_date;
then a primary key integer index may be significantly (2-4 times) faster then an index on a date column.
Postgres supports to some extend clustered indexes, which is what you suggest by removing and reinserting the data.
In fact, removing and reinserting the data in the order you want will not change the time the query takes. Postgres does not know the order of the data.
If you know that the table's data does not change. Then cluster the data based on the index you create.
This operation reorders the table based on the order in the index. It is very effective until you update the table. The syntax is:
CLUSTER tableName USING IndexName;
See the manual for details.
I also recommend you use
explain <query>;
to compare two queries, before and after an index. Or before and after clustering.
Hi there,
we are setting up a Database in PostgreSQL. And in this DB we have to save a Value for every 15 min of every customer of ours. This value has to do with electic grid management, meaning it is exactly 1 value for every customer (well metering instance) for every 15min intervall (-96 per day - 35040 for every non leap Year).
As of right now we are getting these Values from a CSV Dokument per Customer that is updated every 15min. In the Future we will get this data as an XML for every 15min.
We are looking at partioning and indexing the data we get, to make the query faster.
So there will be a Table with the metadata that refers to the Value table. Now my colleague thinks we should just index the "value"-table by the timestamp, but i think this is not the optimal way. As far as i understand indexing over the timestamp would lead Postgres to rebalance the b-tree for every insert, since every insert call gives us a bunch of new values (1 value per metering instance...)
I would rather configure a custom b+-tree that suits our current and near future sitiuation. this way we do not have a lot of rebalancing.
Also it does not sit right with me that the "resolution" of a timestamp(0) is 3600 per hour instead of the necessary resolution of 4 per hour.
Another thing that bothers me, is values might be misshandeld somewhere in the APIs and the timestamp will change ever so slightly away from the exact time (e.g. 12:15:00 to 12:15:01) due to JS problems with 1/10 values or just plain human errors and this triggers a unexpected rebalancing.
As of now we have around 2000 customers for that table but are expecting to grow to around 50 000 within the next 3 years.
We are both fairly new to Postgres as you can properly tell and would really appreciate some pointers in this disscusion of ours.
I have the following query:
SELECT
COUNT(*)
FROM
users
INNER JOIN
roundtables
ON roundtables.author_id = users.id
AND roundtables.created_at
BETWEEN '2021-06-01' AND '2021-06-18'
AND roundtables.deleted_at IS null
WHERE
users.role = 'teacher'
AND users.deleted_at IS null
;For the date on the example, Postgres generates the following plan:
Aggregate (cost=13120.87..13120.88 rows=1 width=8) (actual time=46.365..46.365 rows=1 loops=1)
Buffers: shared hit=5239 read=869
-> Nested Loop (cost=5048.79..13120.57 rows=120 width=0) (actual time=41.081..46.265 rows=1182 loops=1)
Buffers: shared hit=5239 read=869
-> Bitmap Heap Scan on roundtables (cost=5048.37..8750.23 rows=1098 width=16) (actual time=41.066..42.303 rows=1182 loops=1)
Recheck Cond: ((created_at >= '2021-06-01 00:00:00+00'::timestamp with time zone) AND (created_at <= '2021-06-18 00:00:00+00'::timestamp with time zone) AND (deleted_at IS NULL))
Heap Blocks: exact=1130
Buffers: shared hit=1134 read=869
-> Bitmap Index Scan on roundtables_search_idx (cost=0.00..5048.09 rows=1098 width=0) (actual time=40.948..40.948 rows=1182 loops=1)
Index Cond: ((created_at >= '2021-06-01 00:00:00+00'::timestamp with time zone) AND (created_at <= '2021-06-18 00:00:00+00'::timestamp with time zone) AND (deleted_at IS NULL))
Buffers: shared hit=4 read=869
-> Index Only Scan using users_search_idx on users (cost=0.42..3.97 rows=1 width=16) (actual time=0.003..0.003 rows=1 loops=1182)
Index Cond: ((id = roundtables.author_id) AND (role = 'teacher'::enum_users_role) AND (deleted_at IS NULL))
Heap Fetches: 0
Buffers: shared hit=4105
Planning time: 0.270 ms
Execution time: 46.401 msIf I change the begin date to '2019-06-01', for example, it changes the plan to this:
Aggregate (cost=56965.56..56965.57 rows=1 width=8) (actual time=259.748..259.748 rows=1 loops=1)
Buffers: shared hit=17612 read=22067, temp read=360 written=358
-> Hash Join (cost=24578.81..56938.26 rows=10921 width=0) (actual time=58.842..250.845 rows=100698 loops=1)
Hash Cond: (roundtables.author_id = users.id)
Buffers: shared hit=17612 read=22067, temp read=360 written=358
-> Seq Scan on roundtables (cost=0.00..30543.41 rows=100222 width=16) (actual time=0.070..133.741 rows=100711 loops=1)
Filter: ((deleted_at IS NULL) AND (created_at >= '2019-06-01 00:00:00+00'::timestamp with time zone) AND (created_at <= '2021-06-18 00:00:00+00'::timestamp with time zone))
Rows Removed by Filter: 21183
Buffers: shared hit=6648 read=22067
-> Hash (cost=23330.01..23330.01 rows=71824 width=16) (actual time=58.262..58.262 rows=71669 loops=1)
Buckets: 131072 Batches: 2 Memory Usage: 2718kB
Buffers: shared hit=10964, temp written=156
-> Index Only Scan using users_search_idx on users (cost=0.42..23330.01 rows=71824 width=16) (actual time=0.008..37.979 rows=71669 loops=1)
Index Cond: ((role = 'teacher'::enum_users_role) AND (deleted_at IS NULL))
Heap Fetches: 0
Buffers: shared hit=10964
Planning time: 0.306 ms
Execution time: 259.782 msSo it changes what used to be a bitmap index scan + bitmap heap scan to a sequential scan on the roundtables table. I'm assuming that this is happening because the planner assumes its quicker than doing the bitmap scans.
Ideally, I would like to convert the roundtables scan to an index only scan. Is this possible, and if so, how can I achieve this?
The indexes that I am currently using are the following:
CREATE INDEX users_search_idx ON users(id, role, deleted_at); CREATE INDEX roundtables_search_idx ON roundtables(author_id, created_at, deleted_at);
Don’t worry, be happy
how does postgres perform the index on the timestamp - in other words will it index the timestamp based on the date/ time, or will it go into the seconds/ milliseconds, etc?
The internals of the indexing scheme used by Postgres should generally be transparent to you, of no regard. And keep in mind that the implementation you study today may change in a future version of Postgres.
You are likely falling into the trap of premature optimization. Trust Postgres and its default behaviors until you know you have a demonstrable performance problem.
Moments
Date-time handling is more complicated than you might understand.
Firstly, you are using TIMESTAMP which is actually the abbreviated name for TIMESTAMP WITHOUT TIME ZONE. This type cannot represent a moment. This type stores only a date and a time-of-day. For example, January 23 2020 at 12:00 noon. But does that mean noon in Tokyo Japan? Or noon in Paris France, several hours later? Or noon in Toledo Ohio US, several more hours later?
I suggest always expanding the type name out fully to be very clear in your SQL. Use TIMESTAMP WITHOUT TIME ZONE rather than TIMESTAMP.
But if you are actually trying to represent moments, a specific point on the timeline, you must use the TIMESTAMP WITH TIME ZONE. This name comes from the SQL Standard. But in Postgres, and some other databases, it is a bit of a misnomer. Postgres does not actually store the time zone. Instead, Postgres uses any time zone or offset-from-UTC information submitted along with an input to adjust into UTC. The value written to storage is always in UTC. If you care about the original zone name or offset numbers (hours-minutes-seconds), then you need to store that in a second column.
When retrieved from the database, the value comes out in UTC as well. But be aware that some middleware tools insist on applying a default time zone the value after retrieval. While well-intentioned, this anti-feature can cause much confusion. You will have no such confusion when using java.time objects as shown below.
Span-of-time queries
Postgres is storing a moment in UTC, likely as a count from an epoch-reference date-time given that the data type is documented as being an integer of 64 bits (8 octets). According to Wikipedia, Postgres uses an epoch reference of 2000-01-01, presumably the first moment of that date in UTC, 2000-01-01T00:00:00.0Z. We do not have any reason to care about what epoch references is used, but there you go.
The real point is that a date-time value in Postgres is stored simply as a number, a count of microseconds. The timestamp types are not a specific date and a time-of-day as you may be thinking. Your queries certainly may benefit from an index on the timestamp column, but date-oriented (without time-of-day) queries will not benefit specifically. The index is not date-oriented, nor can it be as I will explain next.
Determining a date from a moment requires a time zone. For any given moment, the date varies around the globe by time zone. A few minutes after midnight in Paris France is a new day while still “yesterday” in Montréal Québec.
To query moments by date, you need to determine the first moment of the day, and the first moment of the following day. Then we use the Half-Open approach to defining a span-of-time where the beginning is inclusive while the ending is exclusive. We search for moments that are equal to or later than the beginning while also being before the ending. Tip: another way of saying "equal to or later than the beginning" is "not before".
You are using Java, so you can make use of the industry-leading java.time classes there.
The java.time classes use a resolution of nanoseconds, finer than the microseconds used in Postgres. So you will have no problem loading Postgres values into Java. However, beware of data loss when going the other direction as nanoseconds will be silently truncated to store only microseconds.
When determining the first moment of the day, do not assume the day starts at 00:00:00.0. Some dates in some zones start at another time such as 01:00:00.0. Always let java.time determine the first moment of the day.
ZoneId z = ZoneId.of( "Asia/Tokyo" ) ; // Or `Africa/Tunis`, `America/Montreal`, etc.
LocalDate today = LocalDate.now( z ) ;
ZonedDateTime zdtStart = today.atStartOfDay( z ) ; // First moment of the day.
ZonedDateTime zdtStop = today.plusDays( 1 ).atStartOfDay( z ) ; // First moment of the following day.
Write your Half-Open SQL statement. Do not use the SQL command BETWEEN because it is not Half-Open.
String sql = "SELECT * FROM tbl WHERE event !< ? && event < ? ;" ; // Half-Open query in SQL.
Pass your beginning and ending values to a prepared statement.
Your JDBC driver supporting JDBC 4.2 and later can work with most java.time by using PreparedStatement::setObject & ResultSet::getObject. Oddly, the JDBC spec does not require support for the two most commonly used types: Instant (always in UTC) and ZonedDateTime. These may or may not work your specific driver. The standard does require support for OffsetDateTime, so let's convert to that.
preparedStatement.setObject( 1 , zdtStart.toOffsetDateTime() ) ;
preparedStatement.setObject( 2 , zdtStop.toOffsetDateTime() ) ;
The resulting OffsetDateTime objects passed to the PreparedStatement will carry the offset used by that time zone at that date-time. For debugging, or curiosity, you may want to see those values in UTC. So let's adjust to UTC by extracting a Instant, and then applying an offset of zero hours-minutes-seconds to get a OffsetDateTime carrying an offset of UTC itself.
OffsetDateTime start = zdtStart.toInstant().atOffset( ZoneOffset.UTC ) ;
OffsetDateTime stop = zdtStop.toInstant().atOffset( ZoneOffset.UTC ) ;
Pass to the prepared statement.
preparedStatement.setObject( 1 , start ) ;
preparedStatement.setObject( 2 , stop ) ;
Once these start and stop values arrive at the database server, they will be converted to a number representing a count-from-epoch, a simple integer. Then Postgres performs a simple number comparison. If an index exists on those integer numbers, that index may or may not be utilized as the Postgres query planner sees fit.
If you have relatively small number of rows, and much RAM to cache them, you may not need an index. Perform tests, and use EXPLAIN/ANALYZE to see the real-world performance.
Date column via Java
If you have done the work to prove a performance problem with date-oriented queries, you could add a second column of type DATE. Then index that column, and refer to it explicitly in your date-oriented queries.
When inserting your moment, also include a calculated value for the date as perceived in whatever time zone makes sense to your app. Just be sure to clearly document your intentions, and the specifics of the time zone used in determining the date. Tip: Postgres offers a feature to include a blurb of text as part of your column's definition alongside the column name and its data type.
As a second DATE column is derived from another column, it is by definition redundant and there de-normalized. As a rule, you should consider de-normalizing only as a last resort.
Java code when inserting a value.
String sql = "INSERT INTO tbl ( event , date_tokyo ) VALUES ( ? , ? ) ;" ;
Determine the current moment, and the current moment's date as perceived in time zone Asia/Tokyo.
Instant now = Instant.now() ; // Always in UTC, no need to specify a time zone here.
OffsetDateTime odt = now.atOffset( ZoneOffset.UTC ) ; // Convert from `Instant` to `OffsetDateTime` if your JDBC driver does not support `Instant`.
ZoneId z = ZoneId.of( "Asia/Tokyo" ) ;
ZonedDateTime zdt = now.atZone( z ) ;
LocalDate localDate = zdt.toLocalDate() ; // Extract the date as seen at this moment by people in the Tokyo time zone.
Pass to your prepared statement.
preparedStatement.setObject( 1 , odt ) ;
preparedStatement.setObject( 2 , localDate ) ;
Now you can make date-oriented queries on the date_tokyo column. Index if need be.
Date column via SQL
Alternatively, you could populate that date_tokyo column automatically within Postgres.
Trigger
You could write a trigger that uses date-time functions built into Postgres to determine the date of that moment as seen in the time zone Asia/Tokyo. The trigger could then write the resulting date value into that second column.
Generated value column
Or, with Postgres 12, you can more simply use the new generated columns feature. This new feature does the same work, but without the bother of defining and attaching a trigger. For discussion of this new feature, see:
- New In PostgreSQL 12: Generated Columns
- Generated columns in PostgreSQL 12 by Kirk Roybal
- PostgreSQL 12: generated columns by Daniel Westermann
In Postgres 12, a column with GENERATED ALWAYS AS (…) STORED has its value physically stored, and can be indexed.
Caveat
Crucial to such date-time work is correct information about the current definitions of time zones. Usually this information comes via the tz data maintained by ICANN/IANA.
Both Java and Postgres contain their own copy of tz data.
Politicians around the world have shown a penchant for redefining their time zones, often with little or no warning. So be sure to keep track of changes to time zones you care about. When you update Java or Postgres, you will likely be getting fresh copy of the tz data. But in some cases you may need to manually update either or both environments (Java & Postgres). And your host OS has a tz data copy as well, fyi.
This is a regurgitation of what Percona recommends.
They recommend the
BRIN index
.
I needed this proof for picking up sets of records ordered by timestamptz. Even though the example uses timestamp I use timestamptz.
And my records are chronological and old timestamptz columns are not updated or deleted.
Other columns in only recent records are updated. Older records are untouched.
My table will have a few million records.
You could test your queries. I use pgAdmin.
CREATE TABLE testtab (id int NOT NULL PRIMARY KEY,date TIMESTAMP NOT NULL, level INTEGER, msg TEXT);
create index testtab_date_idx on testtab(date);
"Gather (cost=1000.00..133475.57 rows=1 width=49) (actual time=848.040..862.638 rows=0 loops=1)"
" Workers Planned: 2"
" Workers Launched: 2"
" -> Parallel Seq Scan on testtab (cost=0.00..132475.47 rows=1 width=49) (actual time=832.108..832.109 rows=0 loops=3)"
" Filter: ((date >= '2019-08-08 14:40:47.974791'::timestamp without time zone) AND (date <= '2019-08-08 14:50:47.974791'::timestamp without time zone))"
" Rows Removed by Filter: 2666667"
"Planning Time: 0.238 ms"
"Execution Time: 862.662 ms"
explain analyze select * from public.testtab where date between '2019-08-08 14:40:47.974791' and '2019-08-08 14:50:47.974791';
"Gather (cost=1000.00..133475.57 rows=1 width=49) (actual time=666.283..681.586 rows=0 loops=1)"
" Workers Planned: 2"
" Workers Launched: 2"
" -> Parallel Seq Scan on testtab (cost=0.00..132475.47 rows=1 width=49) (actual time=650.661..650.661 rows=0 loops=3)"
" Filter: ((date >= '2019-08-08 14:40:47.974791'::timestamp without time zone) AND (date <= '2019-08-08 14:50:47.974791'::timestamp without time zone))"
" Rows Removed by Filter: 2666667"
"Planning Time: 0.069 ms"
"Execution Time: 681.617 ms"
create index testtab_date_brin_idx on rm_owner.testtab using brin (date);
explain analyze select * from public.testtab where date between '2019-08-08 14:40:47.974791' and '2019-08-08 14:50:47.974791';
"Bitmap Heap Scan on testtab (cost=20.03..33406.84 rows=1 width=49) (actual time=0.143..0.143 rows=0 loops=1)"
" Recheck Cond: ((date >= '2019-08-08 14:40:47.974791'::timestamp without time zone) AND (date <= '2019-08-08 14:50:47.974791'::timestamp without time zone))"
" -> Bitmap Index Scan on "testtab_date_brin_idx " (cost=0.00..20.03 rows=12403 width=0) (actual time=0.141..0.141 rows=0 loops=1)"
" Index Cond: ((date >= '2019-08-08 14:40:47.974791'::timestamp without time zone) AND (date <= '2019-08-08 14:50:47.974791'::timestamp without time zone))"
"Planning Time: 0.126 ms"
"Execution Time: 0.161 ms"
Update : All the examples I see are like the one described here
WITH cte AS (
SELECT
date1::date,
rank() OVER (ORDER BY date1)
FROM generate_series(date '2014-03-01', CURRENT_DATE + interval '1' month, interval '6 month') g (date1)
),
cteall AS (
SELECT
all_date::date
FROM
generate_series(date '2014-03-01', CURRENT_DATE + interval '1' month, interval ' 1 day') s (all_date)
),
cte3 AS (
SELECT
*
FROM
cteall c1
LEFT JOIN cte c2 ON date1 = all_date
),
cte4 AS (
SELECT
*,
count(rank) OVER w AS ct_str
FROM
cte3
WINDOW w AS (ORDER BY all_date))
SELECT
*,
rank() OVER (PARTITION BY ct_str ORDER BY all_date) AS rank1,
dense_rank() OVER (ORDER BY all_date) AS dense_rank1
FROM
cte4;
Hope it's not intimidating. personally I found cte is a good tool, since it make logic more clearly.
demo
useful link: How to do forward fill as a PL/PGSQL function
If some column don't need, you can simple replace * with the columns you want.
Based on @Mark's answer I wrote this code below, but it's not simpler than the original code.
select s.date,
m.period_index
from
(
select date::date as half_year_start,
rank() over (order by date) as period_index,
coalesce(lead(date::date, 1) over (), CURRENT_DATE) as following_half_year_start
from generate_series(date '2014-03-01', CURRENT_DATE + interval '1' month, interval '6 month') as date
) m
left join
(
select generate_series(date '2014-03-01', CURRENT_DATE, interval '1 day')::date as date
) s
on s.date between m.half_year_start and m.following_half_year_start
;
Usually database optimizers won't chose to use indexes for open-ended ranges, such a updated_at > somedate.
But, in many cases the datatime column won't exceed "now", so you can preserve the semantic of > somedate by converting the condition to a range by using between like this:
where updated_at between somedate and current_timestamp
A between predicate is much more likely to cause the optimizer to chose to use an index.
Please post if this approach improved your query’s performance.
For any given query, the use of an index depends on the cost of using that index compared to a sequential scan
Frequently developers think that because there is an index, a query should run faster, and if a query runs slow, an index is the solution. This is usually the case when the query will return few tuples. But as the number of tuples in the result increases, the cost of using an index might increase.
You are using postgres. Postgres does not support clustering around a given attribute. That means that postgres, when confronted with a range query (of the type att > a and att < b) needs to compute an estimation of the number of tuples in the result (make sure you vacuum your database frequently) and the cost of using an index compared to doing a sequential scan. it will then decide what method to use.
you can inspect this decision by running
EXPLAIN ANALYZE <query>;
in psql. It will tell you if it uses an index or not.
If you really, really want to use the indexes instead of a sequential scan (sometimes it is needed) and you really really know what you are doing, you can change the cost of a sequential scan in the planner constants or disable sequential scans in favor of any other method. See this page for the details:
http://www.postgresql.org/docs/9.1/static/runtime-config-query.html
Make sure you browse the correct version of the documentation.
--dmg