# Top 7 Linux GPU Monitoring and Diagnostic Commands Line Tools

Author: Vivek Gite Last updated: November 24, 2020 0 comments

A video card is a special circuit board that controls what is displayed on a computer monitor. It is also called a graphics processing unit (GPU), which calculates 3D images and graphics for Linux gaming and other usages. Let us see the top 7 Linux GPU monitoring and diagnostic command-line tools to solve issues.
The following tools work on Linux for GPU monitoring and diagnostic purposes and other operating systems such as FreeBSD. The majority of Linux and FreeBSD users these days use Nvidia, Intel, and AMD GPUs.

## Linux GPU Monitoring and Diagnostic Commands Line Tools

We can use the following tools to monitor, diagnostic, and inspect our Linux or *BSD based systems.

### Finding information about GPU on Linux

To get the GPU info simply run:
`sudo lshw -C display -short
lspci -v | more`
Which is output something as follows:

    H/W path               Device        Class          Description
    ===============================================================
    /0/100/1/0                           display        TU117M [GeForce GTX 1650 Mobile / Max-Q]
    /0/100/2               /dev/fb0      display        UHD Graphics 630 (Mobile)

## 1. glmark2 – Stress-testing GPU performance on Linux

glmark2 is an OpenGL 2.0 and ES 2.0 benchmark command-line utility. We can install it as follows:
`$ sudo apt install glmark2`
Now run it as follows:
`$ glmark2`
Then it will begin the test as follows and would stress test your GPU on Linux:

Linux glmark2 test screen

My benchmark result for Nvidia GeForce GTX 1650 running on Ubuntu Linux 20.04 LTS:

    =======================================================
        glmark2 2014.03+git20150611.fa71af2d
    =======================================================
        OpenGL Information
        GL_VENDOR:     NVIDIA Corporation
        GL_RENDERER:   GeForce GTX 1650 with Max-Q Design/PCIe/SSE2
        GL_VERSION:    4.6.0 NVIDIA 450.80.02
    =======================================================
    [build] use-vbo=false: FPS: 4980 FrameTime: 0.201 ms
    [build] use-vbo=true: FPS: 6927 FrameTime: 0.144 ms
    [texture] texture-filter=nearest: FPS: 5144 FrameTime: 0.194 ms
    [texture] texture-filter=linear: FPS: 4979 FrameTime: 0.201 ms
    [texture] texture-filter=mipmap: FPS: 4030 FrameTime: 0.248 ms
    [shading] shading=gouraud: FPS: 6358 FrameTime: 0.157 ms
    [shading] shading=blinn-phong-inf: FPS: 5810 FrameTime: 0.172 ms
    [shading] shading=phong: FPS: 6425 FrameTime: 0.156 ms
    [shading] shading=cel: FPS: 5720 FrameTime: 0.175 ms
    [bump] bump-render=high-poly: FPS: 4772 FrameTime: 0.210 ms
    [bump] bump-render=normals: FPS: 7187 FrameTime: 0.139 ms
    [bump] bump-render=height: FPS: 6724 FrameTime: 0.149 ms
    [effect2d] kernel=0,1,0;1,-4,1;0,1,0;: FPS: 5278 FrameTime: 0.189 ms
    [effect2d] kernel=1,1,1,1,1;1,1,1,1,1;1,1,1,1,1;: FPS: 3649 FrameTime: 0.274 ms
    [pulsar] light=false:quads=5:texture=false: FPS: 5793 FrameTime: 0.173 ms
    [desktop] blur-radius=5:effect=blur:passes=1:separable=true:windows=4: FPS: 2776 FrameTime: 0.360 ms
    [desktop] effect=shadow:windows=4: FPS: 3913 FrameTime: 0.256 ms
    [buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 1555 FrameTime: 0.643 ms
    [buffer] columns=200:interleave=false:update-dispersion=0.9:update-fraction=0.5:update-method=subdata: FPS: 1703 FrameTime: 0.587 ms
    [buffer] columns=200:interleave=true:update-dispersion=0.9:update-fraction=0.5:update-method=map: FPS: 1800 FrameTime: 0.556 ms
    [ideas] speed=duration: FPS: 5480 FrameTime: 0.182 ms
    [jellyfish] : FPS: 4283 FrameTime: 0.233 ms
    [terrain] : FPS: 746 FrameTime: 1.340 ms
    [shadow] : FPS: 4878 FrameTime: 0.205 ms
    [refract] : FPS: 1580 FrameTime: 0.633 ms
    [conditionals] fragment-steps=0:vertex-steps=0: FPS: 5081 FrameTime: 0.197 ms
    [conditionals] fragment-steps=5:vertex-steps=0: FPS: 4556 FrameTime: 0.219 ms
    [conditionals] fragment-steps=0:vertex-steps=5: FPS: 5293 FrameTime: 0.189 ms
    [function] fragment-complexity=low:fragment-steps=5: FPS: 5048 FrameTime: 0.198 ms
    [function] fragment-complexity=medium:fragment-steps=5: FPS: 4602 FrameTime: 0.217 ms
    [loop] fragment-loop=false:fragment-steps=5:vertex-steps=5: FPS: 4744 FrameTime: 0.211 ms
    [loop] fragment-steps=5:fragment-uniform=false:vertex-steps=5: FPS: 4515 FrameTime: 0.221 ms
    [loop] fragment-steps=5:fragment-uniform=true:vertex-steps=5: FPS: 4948 FrameTime: 0.202 ms
    =======================================================
                                      glmark2 Score: 4584
    =======================================================

## 2. glxgears – Simple Linux GPU performance testing tool

It will displays a set of rotating gears and prints out the frame rate at regular intervals. It has become quite popular as basic benchmarking tool for Linux and Unix-like system such as FreeBSD. Install and run it as follows:
`$ apt install mesa-utils
$ glxgears`
The GPU frame rate is measured and published out on the screen every five seconds. The final result will look as follows:

    Running synchronized to the vertical refresh.  The framerate should be
    approximately the same as the monitor refresh rate.
    299 frames in 5.0 seconds = 59.416 FPS
    299 frames in 5.0 seconds = 59.731 FPS
    300 frames in 5.0 seconds = 59.940 FPS
    300 frames in 5.0 seconds = 59.968 FPS
    300 frames in 5.0 seconds = 59.943 FPS
    300 frames in 5.0 seconds = 59.967 FPS
    299 frames in 5.0 seconds = 59.742 FPS
    300 frames in 5.0 seconds = 59.951 FPS
    .....
    ...
    ...

## 3. gpustat – A simple tool to get Nvidia GPU stats on Linux and FreeBSD Unix

It is written in Python and the perfect tool for CLI users, especially ML/AI developers. One can install it as follows using PIP
`$ pip install gpustat
$ pip3 install gpustat`
Run it as follows:
`$ gpustat
$ gpustat -cp`
Here we see name of running process and their PIDs running on Nvidia GPU:

    nixcraft-wks01                         Tue Nov 24 15:46:37 2020  450.80.02
    [0] GeForce GTX 1650 with Max-Q Design | 39'C,  ?? %,   2 % |   962 /  3911 MB | Xorg/2454(100M) Xorg/3504(325M) gnome-shell/3689(181M) firefox/4614(1M) firefox/5036(1M) firefox/5143(1M)

See help:
`$ gpustat -h`

## 4. intel_gpu_top – Displying a top-like summary of Intel GPU usage on Linux

First install the tool, run:
`$ sudo apt install intel-gpu-tools
## CentOS/RHEL/Fedora Linux user try the dnf command ##
$ sudo dnf install intel-gpu-tools`
Fedora, RHEL and CentOS Linux user can use the podman command as follows to install the same:
`$ podman run --rm --priviledged registry.freedesktop.org/drm/igt-gpu-tools/igt:master`
The tool gathers data using perf performance counters (PMU) exposed by i915 and other platform drivers like RAPL (power) and Uncore IMC (memory bandwidth). Run it as follows on Linux system:
`$ sudo intel_gpu_top`

## 5. nvidia-smi – NVIDIA System Management Interface program

The nvidia-smi provides monitoring and management capabilities for each of NVIDIA’s Tesla, Quadro, GRID and GeForce devices from Fermi and higher architecture families. GeForce Titan series devices are supported for most functions with very limited information provided for the remainder of the Geforce brand. NVSMI is a cross platform tool that supports all standard NVIDIA driver-supported Linux and FreeBSD. Install it as follows once Nvidia driver installed on Ubuntu Linux:
`$ apt install nvidia-smi`
Open the terminal and then run:
`$ nvidia-smi -q -g 0 -d UTILIZATION -l 1
$ sudo nvidia-smi
$ nvidia-smi --help`
Here is what we see:

    Tue Nov 24 15:57:43 2020
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  GeForce GTX 165...  Off  | 00000000:01:00.0  On |                  N/A |
    | N/A   40C    P8     3W /  N/A |   1011MiB /  3911MiB |      2%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |    0   N/A  N/A      2454      G   /usr/lib/xorg/Xorg                100MiB |
    |    0   N/A  N/A      3504      G   /usr/lib/xorg/Xorg                357MiB |
    |    0   N/A  N/A      3689      G   /usr/bin/gnome-shell              179MiB |
    |    0   N/A  N/A      4614      G   /usr/lib/firefox/firefox            1MiB |
    |    0   N/A  N/A      5036      G   /usr/lib/firefox/firefox            1MiB |
    |    0   N/A  N/A      5143      G   /usr/lib/firefox/firefox            1MiB |
    |    0   N/A  N/A      6406      G   ...AAAAAAAA== --shared-files      225MiB |
    |    0   N/A  N/A     14462      G   ...AAAAAAAA== --shared-files      131MiB |
    +-----------------------------------------------------------------------------+

## 6. nvtop – NVIDIA GPU top

Another fancy but very useful tool for NVIDIA GPU. It is a ncurses-based GPU status viewer for NVIDIA GPUs similarly to the htop command or top command. We can install it as follows:
`$ apt install nvtop
## RUN the tool ##
$ nvtop`
The following commands are available while in nvtop is on screen:

- `Up` – Select (highlight) the previous process.
- `Down` – Select (highlight) the next process.
- `Left` / `Right` – Scroll in the process row.
- `+` – Sort increasingly.
- `-` – Sort decreasingly.
- `F1` – Select a signal to send to the highlighted process.
- `F2` – Select the field for sorting. The current sort field is highlighted inside the header bar.
- `F3`, `q`, `Esc` – Exit nvtop and return to your shell

## 7. radeontop – Tool to show AMD GPU utilization on Linux

View your AMD GPU utilization, both for the total activity percent and individual blocks on Linux. Install it as follows:
`$ sudo apt install radeontop
$ sudo radeontop`
It works with R600 and up GPUs, even Southern Islands should work fine. Works with both the open source AMD drivers and AMD Catalyst cloused-source drivers:

## Conclusion

You learned about the various Linux GPU commands and tools for monitoring and diagnostic purposes on Linux and BSD-based systems. Let me know if I missed your favorite tool in the comment section below.