You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns. More on reddit.com

I've used Redshift heavily at a couple places now over the past 5 years. In that time I also got to use BigQuery, Snowflake, Hive, Spark, and Athena. It's a little absurd how many options we have at this point, such as Clickhouse, Firebolt, Pinot, Druid, and the list goes on. The point is, Redshift isn't bad, but it's worse in many ways compared to competition. Redshift does have role based access now, but it's implemented strangely and makes it confusing how to use it with groups. Realistically, that should have been prioritized 6-7 years ago when Redshift had already been out for several years. Redshift finally came up with a way to separate storage and compute with RA3 and Spectrum, but it's still not elastic compute like Snowflake where you can just create another cluster with no downtime and no impact on running queries. Redshift has worked really hard on using probabilistic methods and ML to do internal optimizations for query performance and table structure, but sometimes it gets it really wrong and is hard to impossible to debug without just turning those features off (when you even can). Take auto materialized views as an example. They suck. I'm not going into it, but it's a half assed implementation. BigQuery, Snowflake, and Spark all have far more sophisticated query planners and optimizers that are easier to tune. When it comes down to it, taking advantage of sort and dist keys, data types, and compression encodings is by far the best thing you can do with Redshift in order to maximize efficiency, reduce costs, and get the best developer/analyst experience, which just feels antiquated once you start using other options. I don't have Redshift by any means. It's like Airflow to me. I know it really well at this point and that makes me productive with it, but also I use anything else and feel like an idiot wasting time on this stuff. More on reddit.com

Data warehousing. Lots of data where you want to aggregate queries fast. More on reddit.com

Don't use it for what it wasn't intended and you won't have a problem. EG: Don't try to use it for OLTP workloads. It's a columnar data store, and performant when used correctly.

Our enterprise data warehouse is built on top of redshift and it works great!

Things have also dramatically improved with the addition of the data lake approach for data storage and retrieval. Anything that wasn't performant enough on top of Redshift was moved to near-line storage in AWS Lake Formation, and surfaced to clients through redshift spectrum.

Things that needed real-ish™ time data, was eventually migrated to Kafka.

Just like everything else; when all you have is a hammer, everything looks like a nail. Most of the time, the DB is only one tool in your tool belt.

More on reddit.com