Compress your join keys too, make sure it's the same algorithm so it can compare the compressed values. Distribution is by far the most important to get right, making up to 2 orders of magnitude difference. More on reddit.com

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns. More on reddit.com

I've used Redshift heavily at a couple places now over the past 5 years. In that time I also got to use BigQuery, Snowflake, Hive, Spark, and Athena. It's a little absurd how many options we have at this point, such as Clickhouse, Firebolt, Pinot, Druid, and the list goes on. The point is, Redshift isn't bad, but it's worse in many ways compared to competition. Redshift does have role based access now, but it's implemented strangely and makes it confusing how to use it with groups. Realistically, that should have been prioritized 6-7 years ago when Redshift had already been out for several years. Redshift finally came up with a way to separate storage and compute with RA3 and Spectrum, but it's still not elastic compute like Snowflake where you can just create another cluster with no downtime and no impact on running queries. Redshift has worked really hard on using probabilistic methods and ML to do internal optimizations for query performance and table structure, but sometimes it gets it really wrong and is hard to impossible to debug without just turning those features off (when you even can). Take auto materialized views as an example. They suck. I'm not going into it, but it's a half assed implementation. BigQuery, Snowflake, and Spark all have far more sophisticated query planners and optimizers that are easier to tune. When it comes down to it, taking advantage of sort and dist keys, data types, and compression encodings is by far the best thing you can do with Redshift in order to maximize efficiency, reduce costs, and get the best developer/analyst experience, which just feels antiquated once you start using other options. I don't have Redshift by any means. It's like Airflow to me. I know it really well at this point and that makes me productive with it, but also I use anything else and feel like an idiot wasting time on this stuff. More on reddit.com