I use it to capture the tracer of the run. Very useful to identify the performance bottleneck of your training loop and come up with optimizations. It is a bit of a learning curve to master this technique. You need some understanding how GPU and CPU work together (e.g., GPU kernels are async. When does CPU and GPU sync with each other. What are cuda streams. What can be done in parallel by a GPU) Definitely recommend if you need to understand the performance of your training or inference code. Nsight can be an additional tool since it can provide more information compared to the standard profiler This is an example of using trace and profiler to iteratively optimize a model efficiency performance by the pytorch team https://pytorch.org/blog/accelerating-generative-ai/ More on reddit.com

Hi everyone, I was finally able to solve the issue and I wanted to share my solution with others who may be facing a similar problem. The issue was with the memory usage during validation. I discovered that the memory was being accumulated due to the gradients being retained in the computation graph. To resolve this issue, I added the decorator @torch.no_grad() in the validation_step function. This effectively disables gradient computation, freeing up the memory used by the gradients and resolving the memory leak. code: @torch.no_grad() def validation_step(model, compute_loss, batch, device, tgt_tokenizer, metric): src, src_mask, tgt, tgt_mask, tgt_y, seq_len = batch(device=device) output = model(src, src_mask, tgt, tgt_mask) loss = compute_loss(output, tgt, norm=seq_len) output = model.validation_step(src, src_mask, tgt, tgt_mask) preds = torch.argmax(output, dim=-1) preds = tgt_tokenizer.decode(preds) references = tgt_tokenizer.decode(tgt) score = metric.compute(predictions=preds, references=references) return loss, score['bleu'] PS: the explanation paragraph was generated by ChatGPT :) More on reddit.com

I think you should split this into two categories 1) things that literally make your code run faster such as num_workers, cudNN etc and 2) things that lead to faster convergence such as optimizer, learning rate schedule etc

Otherwise it’s a great list.

More on reddit.com

The amount of GPU compute is too small causing the workload to be CPU bound (i.e. the GPU is not being fed work fast enough). CUDA graph is what you want (check out torch.compile reduce-overhead mode since you are using PyTorch). More on reddit.com