Hi Reddit folks, I wanted to share some benchmarking data I recently compiled running upstage_Llama-2-70b-instruct-v2 on two different hardware setups. If you'd like to see the spreadsheet with the raw data you can check out this link.
Hardware Config #1: AWS g5.12xlarge - 4 x A10 w/ 96GB VRAM
Hardware Config #2: Vultr - 1 x A100 w/ 80GB VRAM
A few questions I wanted to answer:

1. How does the inference speed (tokens/s) between these two configurations compare?

2. How does the number of input tokens impact inference speed?

3. How many input tokens can these machines handle before they start to hit OOM?

4. How does 4-bit vs 8-bit quantization affect all of the above?

Why this model?
I chose upstage_Llama-2-70b-instruct-v2 because it's the current #1 performing OS model on HuggingFace's LLM Leaderboard. Also, according to the documentation the model is able to support 10K+ tokens using RoPE which is allowed me to push memory on the machines to the point of OOM.
Why this hardware?
I have some projects I'm working on that will require high performance LLMs and these are the two most common configurations that we're considering. We do most of our cloud work on AWS so the g5.12xlarge is the "go to" option for inference with a model of this size. However, I have been very interested in understanding if there are compelling reasons to go with a 1xA100 setup which AWS doesn't offer.

Text Generation Performance (t/s) vs Input Tokens (t)

This chart shows how Text Generation Performance (t/s) responds to the number of input tokens (t) sent to the model. As expected, more input tokens results in slower generation speed.

GPU Load Performance (MM:SS)
This is a measure of how long it took to load the model into memory. I averaged this across 5 load attempts for each configuration.

Average Text Generation Performance (tokens/second)
Note that these numbers are an average across all text generation attempts for each configuration.

Maximum Context (tokens)
This was a measure of how many input tokens I could pass into the model before getting an OOM exception for each configuration.

Summary
On text generation performance the A100 config outperforms the A10 config by ~11%. I was surprised to see that the A100 config, which has less VRAM (80GB vs 96GB), was able to handle a larger context size before hitting OOM errors. Additionally, it was interesting to see the A10 hardware was much faster at loading the model. I would presume this is because it can parallelize the load across the 4 separate GPUs. Unsurprisingly, 4-bit quantized models were much faster than 8-bit quantized models (almost 2x) and they were able to handle much larger context sizes before OOM.

Hello,

I just take a g5.xlarge with an Amazon Linux 2023 AMI and I struggle to install an nvidia driver, so please I have some questions:

How can I install an nvida driver there ? I konw there is some AMI which comes already with a driver but I don't want thid solution

Which driver should I use ? My purpose is to play with llama2 to do some GenAI (RAG)

Are nvidia drivers free ?

Thanks a lot

I am using gpu instance “g5.16xlarge” to do inferencing for Mixtral-8x7b-Instruct in AWS Sagemaker, but I am getting an error which says “try changing instance type”.

Does anyone know if “g5.16xlarge” is sufficient for Mixtral-8x7b-Instruct? I noticed in AWS Sagemaker JumpStart, the only gpu instance I can select is “g5.48xlarge” for Mixtral-8x7b-Instruct. Does that mean I will need at least “g5.48xlarge” to run Mixtral-8x7b-Instruct in AWS Sagemaker?

Would really appreciate any input on this. Thanks heaps.