Redshift Serverless or Aurora + S3 Tables? Hands-on experiences wanted!
BEWARE Redshift Serverless + Zero-ETL
Is Redshift Serverless same as Redshift?
How do I get the details about a Spectrum Scan Error on an external table on Redshift Serverless? - Stack Overflow
Videos
I'm currently evaluating Redshift Serverless and Aurora + S3 Tables for a data analytics project. Who has hands-on experiences with both options? I'd be very interested in your advice? What are the differences, that I need to be aware of?
Our RDS database finally grew to the point where our Metabase dashboards were timing out. We considered Snowflake, DataBricks, and Redshift and finally decided to stay within AWS because of familiarity. Low and behold, there is a Serverless option! This made sense for RDS for us, so why not Redshift as well? And hey! There's a Zero-ETL Integration from RDS to Redshift! So easy!
And it is. Too easy. Redshift Serverless defaults to 128 RPUs, which is very expensive. And we found out the hard way that the Zero-ETL Integration causes Redshift Serverless' query queue to nearly always be active, because it's constantly shuffling transitions over from RDS. Which means that nice auto-pausing feature in Serverless? Yeah, it almost never pauses. We were spending over $1K/day when our target was to start out around that much per MONTH.
So long story short, we ended up choosing a smallish Redshift on-demand instance that costs around $400/month and it's fine for our small team.
My $0.02 -- never use Redshift Serverless with Zero-ETL. Maybe just never use Redshift Serverless, period, unless you're also using Glue or DMS to move data over periodically.
We have a new request from one of our clients to use Redshift Serverless as the data warehouse. I never used Serverless redshift, I built warehouses with resdhift cluster and designed ETL pipelines using Glue.
Is serverless same as non-serverless? Setting up, ETL connectibility, and everything else is same? Is serverless cost effective? This client usually does not have so much data. so we will be migrating very small data at every 4 hours periodically.