bfloat16 is generally easier to use, because it works as a drop-in replacement for float32. If your code doesn't create nan/inf numbers or turn a non-0 into a 0 with float32, then it shouldn't do it with bfloat16 either, roughly speaking. So, if your hardware supports it, I'd pick that.
Check out AMP if you choose float16.
I'm would love to know what the difference is between bfloat16 and float16. Does anyone know of any difference? I know that older machines may not support bfloat16 but I would love to know more. Does float16 take longer or incompatible with certain gpus?
Why is there such a huge performance gap between bfloat16, float16, and float32?
Difference in SM performance of float16 and bfloat16
Why to keep parameters in float32, why not in (b)float16?
Reflection and the Never-Ending Confusion Between FP16 and BF16
Videos
bfloat16 is generally easier to use, because it works as a drop-in replacement for float32. If your code doesn't create nan/inf numbers or turn a non-0 into a 0 with float32, then it shouldn't do it with bfloat16 either, roughly speaking. So, if your hardware supports it, I'd pick that.
Check out AMP if you choose float16.
Let’s set aside the API drama for a moment. This topic deserves careful consideration, as I keep seeing the same mistake made repeatedly.
The author of Reflection is facing issues with the model uploaded to Hugging Face. After three different uploads, the model on Hugging Face still performs much worse than what the author claims it is capable of. People have tested it, and it is underperforming even compared to the baseline LLaMA 3.1 70B.
I’m not sure if Reflection is a scam or not, but there’s a significant issue with the weights.
-
LLama 3.1 70B was trained using BF16, and the wigths are uploaded in BF16: https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct
-
Reflection 70B was converted into FP16: https://huggingface.co/mattshumer/ref_70_e3
Does this make a difference? Yes, it makes a massive difference. BF16 and FP16 are very different formats, and they are not compatible. You cannot convert a BF16 model to FP16 without losing a lot of information.
FP16 has a 5-bit exponent and a 10-bit mantissa, while BF16 has an 8-bit exponent and a 7-bit mantissa. There is no way to convert a BF16 model to FP16, or vice versa, without significant loss of information. The BF16 to FP16 conversion is especially damaging. FP16 is not suitable for neural networks unless you use a complex mixed-precision training approach (https://arxiv.org/abs/1710.03740). On the other hand, BF16, developed by DeepMind (which stands for Brain Float 16) works out of the box for training neural networks.
FP16 was used in the early days for encoder-only models like BERT and RoBERTa, which were typically run in FP16. However, T5 was released in BF16, and since then, no other major model has used FP16 because it simply doesn’t work well. The only reason FP16 was used in the past is that Nvidia didn’t support BF16 until the A100 GPU came out. Google TPUs, however, had BF16 support, which is why T5 was trained in BF16.
I’m bringing this up because, despite FP16 being a dead format, and BF16 being the format used for every big model, many people still confuse them. This seems to have happened to the author of Reflection. Please, do not use FP16, and above all, do not attempt to convert BF16 weights into FP16, it will ruin your model.
Previous to talking to a colleague I assumed they meant mixed precision using bfloat16, but he told me he regularly trains models using pure bfloat16, as in first transforms the model to bfloat16 and then proceeds with regular model training... is this common?