As stated in Python's docs:

Floating point numbers are usually implemented using double in C

double in C is normally a 64-bit number (double-precision, as opposed to the single-precision 32-bit float type), and it's also known as float64. So, what we call float in plain Python is usually a 64-bit float, or torch.float64.

PyTorch itself, however, uses float32 by default. AFAIK, this is done to save memory: you'll likely be training models with quite a lot of parameters, and using 32-bit floats will take two times less memory than Python's 64-bit float.

Use torch_dtype='auto' in from_pretrained(). Example:

model2 = AutoModelForTokenClassification.from_pretrained(save_directory, 
                                                         local_files_only=True,
                                                         torch_dtype='auto')

Full example:

# pip install transformers
from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

# Load model
model_path = 'huawei-noah/TinyBERT_General_4L_312D'
model = AutoModelForTokenClassification.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Convert the model to FP16
model.half()

# Check model dtype
def print_model_layer_dtype(model):
    print('\nModel dtypes:')
    for name, param in model.named_parameters():
        print(f"Parameter: {name}, Data type: {param.dtype}")

print_model_layer_dtype(model)
save_directory = 'temp_model_SE'
model.save_pretrained(save_directory)

model2 = AutoModelForTokenClassification.from_pretrained(save_directory, local_files_only=True, torch_dtype='auto')
print('\n\n##################')
print(model2)
print_model_layer_dtype(model2)

It'll load model2 as torch.float16.