For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU. There also is a list of compute processes and few more options but my graphic card (GeForce 9600 GT) is not fully supported.
Sun May 13 20:02:49 2012
+------------------------------------------------------+
| NVIDIA-SMI 3.295.40 Driver Version: 295.40 |
|-------------------------------+----------------------+----------------------+
| Nb. Name | Bus Id Disp. | Volatile ECC SB / DB |
| Fan Temp Power Usage /Cap | Memory Usage | GPU Util. Compute M. |
|===============================+======================+======================|
| 0. GeForce 9600 GT | 0000:01:00.0 N/A | N/A N/A |
| 0% 51 C N/A N/A / N/A | 90% 459MB / 511MB | N/A Default |
|-------------------------------+----------------------+----------------------|
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0. Not Supported |
+-----------------------------------------------------------------------------+
Answer from pbm on Stack ExchangeFor Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU. There also is a list of compute processes and few more options but my graphic card (GeForce 9600 GT) is not fully supported.
Sun May 13 20:02:49 2012
+------------------------------------------------------+
| NVIDIA-SMI 3.295.40 Driver Version: 295.40 |
|-------------------------------+----------------------+----------------------+
| Nb. Name | Bus Id Disp. | Volatile ECC SB / DB |
| Fan Temp Power Usage /Cap | Memory Usage | GPU Util. Compute M. |
|===============================+======================+======================|
| 0. GeForce 9600 GT | 0000:01:00.0 N/A | N/A N/A |
| 0% 51 C N/A N/A / N/A | 90% 459MB / 511MB | N/A Default |
|-------------------------------+----------------------+----------------------|
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0. Not Supported |
+-----------------------------------------------------------------------------+
For linux, use nvidia-smi -l 1 will continually give you the gpu usage info, with in refresh interval of 1 second.
[seedship@triple-destinies ~]$ nvidia-smi
Tue Dec 29 22:31:07 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01 Driver Version: 455.45.01 CUDA Version: 11.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1070 Off | 00000000:01:00.0 On | N/A |
| N/A 64C P0 55W / N/A | 1134MiB / 8116MiB | 79% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 770 G /usr/lib/Xorg 322MiB |
| 0 N/A N/A 128937 G /usr/lib/firefox/firefox 1MiB |
| 0 N/A N/A 139902 G ...mviewer/tv_bin/TeamViewer 207MiB |
| 0 N/A N/A 147796 G ...AAAAAAAA== --shared-files 26MiB |
| 0 N/A N/A 148489 G ...e/Steam/ubuntu12_32/steam 70MiB |
| 0 N/A N/A 148504 G ./steamwebhelper 1MiB |
| 0 N/A N/A 148516 G ...token=1395340159186027037 434MiB |
| 0 N/A N/A 149878 G ...token=1571661428272191793 55MiB |
| 0 N/A N/A 150371 G /usr/lib/firefox/firefox 3MiB |
+-----------------------------------------------------------------------------+I noticed high GPU usage on my GPU from nvidia-smi, but I saw that there are so many tasks utilizing GPU, I don't know who is causing it to be that high. I searched online for GPU profilers, but all I saw was nvidia-smi.
Does anyone know if there is a way to break down the % utilization of each task?
A top-like utility for monitoring CUDA activity on a GPU - Stack Overflow
Monitor GPU usage with nvidia-smi
Check NVIDIA GPU Usage
One small aspect where Linux is really lacking is GPU resource monitoring
Videos
For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU.
For Intel GPU's you can use the intel-gpu-tools.
AMD has two options
fglrx (closed source drivers):
aticonfig --odgc --odgtAnd for mesa (open source drivers), you can use RadeonTop
sudo apt install radeontop radeontop
Source:GPU usage monitoring
You can use gpustat, which is a simple command-line script (wrapper for nvidia-smi) for querying and monitoring GPU status:

To get real-time insight on used resources, do:
nvidia-smi -l 1
This will loop and call the view at every second.
If you do not want to keep past traces of the looped call in the console history, you can also do:
watch -n0.1 nvidia-smi
Where 0.1 is the time interval, in seconds.

I find gpustat very useful. It can be installed with pip install gpustat, and prints breakdown of usage by processes or users.
