Hi all,
I have a business request to ETL data from several sources into a data warehouse to run BI tools on (Tableau or PowerBI for example).
The dataset isn't that big, 400GB growing 4GB per week. I can't find any good explanation on if I should use RDS or Redshift to store the data.
Can anyone with experience on this topic guide me please?
Thank you!
amazon web services - Difference between RDS and Redshift - Stack Overflow
amazon web services - Is AWS Redshift to PostgreSQL the same as AWS Aurora to MySQL? - Stack Overflow
AWS RDS, Amazon Athena, Amazon QuickSight and Amazon Redshift
Deciding between RDS and Redshift for "data warehouse"
Can DynamoDB be used by applications running on any operating system?
What type of indexing is used in DynamoDB?
What about database encryption on AWS?
RDS is a managed service for Online Transaction Processing databases (OLTP), i.e. a managed service for the usual MySQL, PostgreSQL, Oracle, MariaDB, Microsoft SQL Server or Aurora (Amazon's own relational database)
Redshift is a managed service for data warehousing, i.e. columnar oriented storage, typical for business analytics type of workloads.
It depends on your usecase.
Put it in example:
If you use Select * from <table> more often, go for RDS. (OLTP)
If you use features like SUM(A) more often, go for RedShift. (OLAP)
Redshift is not PostgreSQL. It is a column store engine that uses a very heavily modified part of a very old PostgreSQL version as its front-end. Under the hood it's powered by ParAccel, a very heavily modified fork of PostgreSQL 8.0.2.
Imagine someone took MySQL 4.1 or something from that era, deleted InnoDB and MyISAM, added their own hardwired storage engine, removed a whole bunch of features and added a bunch of different ones - changing the supported SQL dialect in the process. That gives you some idea.
It's a dramatically different product for different needs. It's heavily optimised for OLAP workloads and pays a heavy price for OLTP workloads.
In general you should use PostgreSQL (on AWS RDS, or elsewhere) for your day to day transaction processing. If you want data warehousing and analytics and have outgrown PostgreSQL for that then you might consider Redshift as one of your options... though it's likely you haven't really outgrown PostgreSQL, just AWS RDS.
Maybe you're looking for something more like Postgres-XL ?
The other answer is accurate regarding Redshift not being the PostgreSQL equivalent of Aurora. Generally you'd use Redshift when you needed to run some very heavy queries on a large dataset (the stuff that might take hours or more to finish running). Redshift is a columnar datastore that essentially auto-normalizes every piece of data that comes in and can execute queries that would otherwise take days in seconds. When you're done, you delete it and then repeat the process when you need it again.
In terms of getting an Aurora equivalent for PostgreSQL, I don't know how far off that is but I'm pretty sure an enterprising person could build their own with AWS EFS (https://aws.amazon.com/efs/). I'm fairly certain that's a big part of the Aurora formula.