amazon web services - Which is lower cost, Sagemaker or EC2? - Stack Overflow
Suggest instance type NVIDIA GPU has the best cost
Benchmarking Inexpensive AWS Instances
Sagemaker pricing and spot instances
I've listed some pros and cons from experience...
..., as opposed to marketing materials. If I were to guess, I'd say you have a much higher chance to experience all the drawbacks of SageMaker, than any one of the benefits.
Drawbacks
- Cloud vendor lock in: free improvements in the open source projects in the future and better prices in competitor vendors are difficult to get. Why don't AWS invest developers in JupyterLab, they have done limited work in open source. Find some great points here, where people have experienced companies using as few AWS services as possible with good effect.
- SageMaker instances are currently 40% more expensive than their EC2 equivalent.
- Slow startup, it will break your workflow if every time you start the machine, it takes ~5 minutes. SageMaker Studio apparently speeds this up, but not without other issues. This is completely unacceptable when you are trying to code or run applications.
- SageMaker Studio is the first thing they show you when you enter SageMaker console. It should really be the last thing you consider.
- SageMaker Studio is more limited than SageMaker notebook instances. For example, you cannot mount an EFS drive.I spoke to a AWS solutions architect, and he confirmed this was impossible (after looking for the answer all over the internet). It is also very new, so there is almost no support on it, even by AWS developers.
- Worsens the disorganised Notebooks problem. Notebooks in a file system can be much easier to organise than using JupyterLab. With SageMaker Studio, a new volume gets created and your notebooks lives in there. What happens when you have more than 1...
- Awful/ limited terminal experience, coupled with tedious configuration (via Lifecycle configuration scripts, which require the Notebook to be turned off just to edit these scripts). Additionally, you cannot set any lifecycle configurations for Studio Notebooks.
- SageMaker endpoints are limited compared to running your own server in an EC2 instance.
- It may seem like it allows you to skip certain challenges, but in fact it provides you with more obscure challenges that no one has solved. Good luck solving them. The rigidity of SageMaker and lack of documentation means lots of workarounds and pain. This is very expensive.
Benefits
These revolve around the SageMaker SDK (the Sagemaker console and SageMaker SDK) (please comment or edit if you found any more benefits)
- Built in algorithms (which you can easily just import in your machine learning framework of choice): I would say this is worse than using open source alternatives.
- Training many models easily during hyperparameter search YouTube video by AWS (a fast way to spend money)
- Easily create machine learning related AWS mechanical turk tasks. However, mturk is very limited within SageMaker, so youre better off going to mturk yourself.
My suggestion
If you're thinking about ML on the cloud, don't use SageMaker. Spin up a VM with a prebuilt image that has PyTorch/ TensorFlow and JupyterLab and get the work done.
You are correct about EC2 being cheaper than Sagemaker. However you have to understand their differences.
- EC2 provides you computing power
- Sagemaker (try to) provides a fully configured environment and computing power with a seamless deployment model for you to start training your model on day one
If you look at Sagemaker's overview page, it comes with Jupyter notebooks, pre-installed machine learning algorithms, optimized performance, seamless rollout to production etc.
Note that this is the same as self-hosting a EC2 MYSQL server and utilizing AWS managed RDS MYSQL. Managed services always appears to be more expensive, but if you factor in the time you have to spent maintaing server, updating packages etc., the extra 30% cost may be worth it.
So in conclusion if you rather save some money and have the time to set up your own server or environment, go for EC2. If you do not want to be bothered with these work and want to start training as soon as possible, use Sagemaker.
I recently did some testing using Dolphin-Llama3 across various (inexpensive-ish) AWS instances to compare performance. The results are in line with what one might expect.
Testing was done using default settings with Ollama. I spun up a new instance on Ubuntu, installed Ollama and ran it with Dolphin-Llama3 —verbose.
Key Takeaways:
-Fastest Prompt Eval Rate: AWS g5 (fastest AWS instance tested)
-Fastest Eval Rate: Home PC w/RTX 3080
-Best Cost-Performance Balance: AWS g4dn.xlarge offers a good balance of performance and cost, at $0.58/hr.
-GPU speed is the key differentiator. Within the same family of models, such as the g4dn and g5 instances, the evaluation rates remain consistent. If the model fits in GPU memory there is no need for more cores/memory.
-I did notice that the more system memory available the greater number of tokens used in the output.
Test Results
AWS Instances
c7g.8xlarge (Compute Instance) •32 cores, 64GB RAM •Prompt Eval Rate: 38.38 tokens/s •Eval Rate: 25.07 tokens/s •Price: $1.27/hr, $941.16/mo r6g.4xlarge (Memory Instance) •16 cores, 128GB RAM •Prompt Eval Rate: 10.15 tokens/s •Eval Rate: 8.29 tokens/s •Price: $0.88/hr, $657.10/mo g4dn.xlarge (GPU Instance) •4 cores, 16GB RAM, 16GB GPU •Prompt Eval Rate: 222.23 tokens/s •Eval Rate: 41.71 tokens/s •Price: $0.58/hr, $434.50/mo g4dn.2xlarge (GPU Instance) •8 cores, 32GB RAM, 32GB GPU •Prompt Eval Rate: 214.25 tokens/s •Eval Rate: 41.74 tokens/s •Price: $0.84/hr, $621.24/mo g5.xlarge (GPU Instance) •4 cores, 16GB RAM, 24GB GPU •Prompt Eval Rate: 624.29 tokens/s •Eval Rate: 68.08 tokens/s •Price: $1.12/hr, $831.05/mo g5.2xlarge (GPU Instance) •8 cores, 32GB RAM, 24GB GPU •Prompt Eval Rate: 624.48 tokens/s •Eval Rate: 66.67 tokens/s •Price: $1.35/hr, $1,000.96/mo
Local Machines
M2 MacMini •M2, 8GB RAM, <8GB GPU •Prompt Eval Rate: 66.38 tokens/s •Eval Rate: 18.33 tokens/s M1 MacBook Air •M1, 16GB RAM, <16GB GPU •Prompt Eval Rate: 71.58 tokens/s •Eval Rate: 11.46 tokens/s Home PC w/RTX 3080 •Intel i5, 64GB RAM, 10GB GPU •Prompt Eval Rate: 185.67 tokens/s •Eval Rate: 83.79 tokens/s
Oracle Ampere
Ampere 16 Core, 32GB RAM •Prompt Eval Rate: 11.96 tokens/s (Duration: 1m34.955180835s) •Eval Rate: 9.01 tokens/s (Duration: 1m28.461256s) •Price: $0.1276/hr, $95/mo Ampere 32 Core, 32GB RAM •Prompt Eval Rate: 22.54 tokens/s (Duration: 47.93207936s) •Eval Rate: 14.11 tokens/s (Duration: 44.423782s) •Price: $0.2796/hr, $208/mo
Here's the data formatted in table for easier viewing - courtesy of u/sergeant113. https://www.reddit.com/r/LocalLLaMA/comments/1dclmwt/comment/l7zrgzm/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
I've been looking into whether it would be sensible to use AWS Sagemaker for my GPU inference workload on a T4 GPU. It can tolerate multiple minutes of downtime and delays, and has large binary multimedia files as input to the model. Autoscaling is a requirement based on load.
AWS Sagemaker Asynchronous endpoints look very attractive in that they manage the container orchestration, autoscaling, queueing of requests and provide various convenience utilities for maintaining, monitoring and upgrading models in production.
In us-east-1, I calculate the following:
| Service | Instance Type | Hourly Price | Monthly Price |
|---|---|---|---|
| EC2 On Demand | g4dn.xlarge | $0.526 | $378 |
| EC2 Annual Reservation | g4dn.xlarge | $0.309 (upfront) | $229 |
| EC2 Spot | g4dn.xlarge | ~$0.22 | ~$156 |
| Sagemaker On Demand | ml.g4dn.xlarge | $0.7364 | $530 |
| Sagemaker On Demand with Saving Plan | ml.g4dn.xlarge | $0.4984 | $358 |
From what I can see however, it would come with a significant cost (likely prohibitively high) to use Sagemaker versus using EKS/ECS to host the model and SQS to provide the queueing of requests. I appreciate that's the price one pays for a managed service, but I wanted to confirm a few things with the community to make sure I'm not missing anything in my cost estimations:
Is it correct that Sagemaker does not support spot instances for inference at all? (I appreciate they support it for training)
Is it correct that one can apply a savings plan to inference endpoints and that it would be Service classified as "Hosting" on this page https://aws.amazon.com/savingsplans/ml-pricing/ ? It's confusing as "Hosting Service" is not a term they use in the development docs to describe inference endpoints per say.
Is it correct one cannot reserve instances for a year for Sagemaker like with EC2 to cut costs, and thus the above Savings Plan is the cheapest you can get a T4 GPU.
I ask this as it seems surprising there's ~40% markup in cost for using this managed service versus EC2, and despite what the AWS Report on TCO says, I can't quite see it saving me that amount of money versus us setting up a EKS/ECS solution for this problem. I can see however that TCO report is also largely considering training infra, which indeed does likely bring a lot of value not relevant here.