I’ve been working with DynamoDB for some time now, but I’m wondering if moving data to Redshift would make more sense for complex queries and analytics. Has anyone else made this switch? do you think it’s worth the effort for better performance and scalability?
Hey I’m a newly graduated student, who started a SaaS, which is now at $5-6k MRR.
When is the right time to move from DynamoDB to a more structured database like Aurora or RDS?
When I was building the MVP I was basically rushing and put everything into DynamoDB in an unstructured way (UserTable, things like tracking affiliate codes, etc).
It all functions perfectly and costs me under $2 per month for everything. The fact of this is really attractive to me - I have around 100-125 paid users and over the year have stored around 2000-3000 user records in dynamoDB. — it doesn’t make sense to just got to a $170 Aurora monthly cost.
However I’ve recently learned about SQL and have been looking at Aurora but I also think at the same time it is still a bit overkill to move my back end databases to SQL from NoSQL.
If I stay with DynamoDB, are there best practices I should implement to make my data structure more maintainable?
This is really a question on semantics and infrastructure - the dynamoDB does not have any performance and I really like the simplicity, but I feel it might be causing some more trouble?
The main things I care about is dynamic nature and where I can easily change things such as attribute names, as I add a lot of new features each month and we are still in the “searching” phase of the startup so lots of things to change - the plan, is to not really have a plan, and just follow customer feedback.
Hi all,
I'm developing a monitoring software which stores its data in a mysql database. One of the tables looks like this:
-
time - (int - unixtimestamp)
-
monitor_id (int - foreign key)
-
value (int)
I will have hundreds/thousands (might event be hundreds of thousands) of nodes reporting back hundreds of rows every minute. This table will probably contain > 99.99% of all my data in the database.
This data is later joined with other tables (with monitor_id which is a primary key in another table) in the same database and used to create charts and other stuff.
My question to you all is that should I keep this table in this database and go with Aurora, or should I consider using DynamoDB or maybe even RedShift? I dont want the most expensive solution but I don't want to be cheap either. I just want to make sure that I've considered the right tools for the right job.
What are your thoughts?
If you're looking for a managed service, have you considered Athena? Dump the data to S3 and run your queries against that directly?
Of the 3 options Dynamo with a composite primary key would be what I looked at first but if you're expecting hundreds of writes per node per minute the IO costs might be prohibitive unless you do something like buffer into redis (or even SQS) and write in batches as near 4KB as possible.
I currently have a usecase where API gateway writes processing jobs to an SQS queue, an ASG scales up and workers pull, process, and delete messages from queue and write to some output queue. I needed to use ASG here since we're processing with an older legacy app which needs Windows and high-CPU (Fargate wouldn't work here).
As the system has started getting much more use, I wanted to log metadata somewhere. Just general things like process ID, input parameters, timestamps, and user info.
I already implemented thus with DynamoDB since it's fast and serverless, and have a lambda on both pre and post processing queues that wrote metadata to the table. For now DynamoDB was very quick to get going, and the only negative I see is that it'll be harder to change the partition and sort keys if we decide to down the line.
However, it got me thinking when should I lean towards RDS and when should I lean towards DynamoDB? Especially if a coworker asks for justification for either architecture, it would be good to weigh the pros ans cons.
I am preparing data engineer exam. Can I please ask the general data distribution rule for Redshift and DynamoDB? Can I please ask why DynamoDB need GSI and LSI? Do operational overhead represent costs and effort of infrastructure management? Thank you so much in advance
I am building a web app that should quickly retrieve and display some data from a table that has over 30 million records.
The data is a result of some ETL that is happening in a Redshift cluster. Some colleagues have suggested that I should use DynamoDB because it is fast and can respond well to features we might add down the road.
However, I don't know a good way to dump the entire 30 million row Redshift table to DynamoDB and I am worried that continually updating the entire table will be expensive due to all the writes.
Can anyone give me some guidance on whether DynamoDB is a good idea or I should push back?
It all depends on how you are going to use the data.
First, from a volume perspective, 30 million rows can easily be handled in pretty much any DB that you want. I’ve done ~1 billion rows in a regular Postgres table just fine.
How do you need to access the data? Are you always going to be doing a lookup based on a known key? Because DynamoDB is mostly a key-value store. If you want to do lookups based on ‘where’ clauses (e.g. select *from table where name like ‘%smith%’ and value < 100), then a traditional DB would be better.
DynamoDB is also schema less. Everything is stored as a JSON blob. This is either good, or bad, depending on what you need.
Pricing in Dynamo is based on the rate of access. So, if you need to read (or write) all 30 million rows in, say, 1 second, that’s going to be expensive. If you architect the solution so that you can do the work over many hours, then it will be cheap.
Is there a reason you can’t directly access the data from redshift?
Could also consider materialized views. They’re presently in preview.
https://docs.aws.amazon.com/redshift/latest/dg/mv.html
For comparison's sake --- BigQuery and Redshift have market shares of 13% vs. 17% in their category, respectively.
Meanwhile, Bigtable's is 1% vs. DynamoDB's 10%.
Thanks!
I heard that even at AWS, all newly services/products are built on DDB unless they have a compelling reason not to because
Of the maintainability, performance and scalability benefits that DDB offers
With that in mind, when would you want to use RDS over DynamoDB?
If you need joins that are offered in SQL, you could define multiple tables in DDB and load from those tables as needed.
One other thing I noticed was that RDS has a set cost whereas in DDB, you pay for WCU and RCU usage
Hello all, I have a JavaScript application I'm developing and running into a bit of architecture decision. The application will accept some form values that would need to be written to either a bucket/warehouse/database. After I will run a python script against the collected data to find matching values/patterns. Would I be better off having everything sit in S3 or Redshift or possibly something else? I am thinking about this from not only a functional aspect but also cost.
You shouldn't be thinking about using Redshift as I don't think that's the right tool for the job.
Redshift is for analyzing tens, hundreds, or thousands of gigabytes of data.
I suggest you look at Dynamodb as your use case doesn't sound like you need the features (and complexity) of a relational database (aka RDS): https://aws.amazon.com/dynamodb/
RDS. Just a standard relational db.
Certainly not redshift for just a handful of tables
We have couple of tables that needs to be made available to redshift for further analysis. I was hoping that dynamodb -> dynamodb stream -> redshift would do the trick but i couldn't find a single tutorial which does this job. everything that i could find involves some lambdas or AWS Glue with custom code.
Is there a way to push dynamodb table data to redshift on regular basis? (assuming there is no data transformation needed) ?
I don’t know why but I have a hard time wrapping my head around data bases. Most of the other services I’m fairly comfortable with.
Edit: Also to add on would appreciate some recommendations for white papers on how particular companies shifted their work loads to the cloud. It’s cool to see how architects implement stuff.
Not sure about whitepapers but this video is great: https://www.youtube.com/watch?v=-pb-DkD6cWg
Let me try to put how i see it in plain english:
Relational database in general => probably Amazon Aurora
Specific relational database engine and version => Amazon RDS
Non-relational low-latency high-scale => Amazon DynamoDB
In-memory cache => Amazon Elasticache
In-memory cache for DynamoDB only => DynamoDB DAX
High-scale analytics / data warehousing => Amazon Redshift
Analytics on top of S3 Data => Amazon Athena
Analytics on top of S3 Data if already using Redshift => Redshift Spectrum
Documents with MongoDB Compatibility => DocumentDB
Search indexing => Amazon Elasticsearch Service
Imutable and cripto verifiable => QLDB
Time series database => Timestream (preview)
also, this video goes in depth about how Robinhood migrated their data warehouse to AWS
Hi all,
I have a business request to ETL data from several sources into a data warehouse to run BI tools on (Tableau or PowerBI for example).
The dataset isn't that big, 400GB growing 4GB per week. I can't find any good explanation on if I should use RDS or Redshift to store the data.
Can anyone with experience on this topic guide me please?
Thank you!
How would we best do the following
-
Copy a DynamoDB table to Redshift
-
Backup the DynamoDB table to S3
-
Delete the DynamoDB table
Would Data Pipeline do all this for us on a schedule? Or could we do all via Lambda and the CLI?
Redshift can copy from dynamodb directly (though I've never had need to test this) http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/RedshiftforDynamoDB.html
There's a DataPipeline template for exporting a DDB table to S3, which is basically your second requirement. You can then load from S3 to Redshfit using COPY, which will probably be faster and cheaper than going through EMR, and definitely much more so than Lambda/CLI since the data moves directly from S3 into Redshift.