Demonstrations of biosnoop, the Linux BPF/bpftrace version.


This traces block I/O, and shows the issuing process (at least, the process
that was on-CPU at the time of queue insert) and the latency of the I/O:

# ./biosnoop.bt
Attaching 4 probes...
TIME(ms)     DISK             COMM             PID    LAT(ms)
611          nvme0n1          bash             4179        10
611          nvme0n1          cksum            4179         0
627          nvme0n1          cksum            4179        15
641          nvme0n1          cksum            4179        13
644          nvme0n1          cksum            4179         3
658          nvme0n1          cksum            4179        13
673          nvme0n1          cksum            4179        14
686          nvme0n1          cksum            4179        13
701          nvme0n1          cksum            4179        14
710          nvme0n1          cksum            4179         8
717          nvme0n1          cksum            4179         6
728          nvme0n1          cksum            4179        10
735          nvme0n1          cksum            4179         6
751          nvme0n1          cksum            4179        10
758          nvme0n1          cksum            4179        17
783          nvme0n1          cksum            4179        12
796          nvme0n1          cksum            4179        25
802          nvme0n1          cksum            4179        32
[...]

This output shows the cksum process was issuing block I/O, which were
completing with around 12 milliseconds of latency. Each block I/O event is
printed out, with a completion time as the first column, measured from
program start.


An example of some background flushing:

# ./biosnoop.bt
Attaching 4 probes...
TIME(ms)     DISK             COMM             PID    LAT(ms)
2966         nvme0n1          jbd2/nvme0n1-8   615          0
2967         nvme0n1          jbd2/nvme0n1-8   615          0
[...]


There is another version of this tool in bcc: https://github.com/iovisor/bcc
The bcc version provides more fields.
