First of all, your test case of using sleep and page-faults is not an ideal test case. There should be no page fault events during the sleep duration, you you can't really expect anything interesting. For the sake of easier reasoning I suggest to use the ref-cycles (hardware) event and a busy workload such as awk 'BEGIN { while(1){} }'.

Question 1: It is my understanding that perf stat gets a "summary" of counts but when used with the -I option gets the counts at the specified millisecond interval. With this option does it sum up the counts over the interval or get the average over the interval, or something else entirely? I assume it is summed up.

Yes. The values are just summed up. You can confirm that by testing:

$ perf stat -e ref-cycles -I 1000 timeout 10s awk 'BEGIN { while(1){} }'
#           time             counts unit events
 1.000105072      2,563,666,664      ref-cycles                                                  
 2.000267991      2,577,462,550      ref-cycles                                                  
 3.000415395      2,577,211,936      ref-cycles                                                  
 4.000543311      2,577,240,458      ref-cycles                                                  
 5.000702131      2,577,525,002      ref-cycles                                                  
 6.000857663      2,577,156,088      ref-cycles                                                  

[ ... snip ... ]
[ Note that it may not be as nicely consistent on all systems due dynamic frequency scaling ]

$ perf stat -e ref-cycles -I 3000 timeout 10s awk 'BEGIN { while(1){} }' 
#           time             counts unit events
 3.000107921      7,736,108,718      ref-cycles                                                  
 6.000265186      7,732,065,900      ref-cycles                                                  
 9.000372029      7,728,302,192      ref-cycles     

Question 2: Why doesn't perf stat -e <event1> -I 1000 sleep 5 give about the same counts as if I summed up the counts over each second for the following command perf record -e <event1> -F 1000 sleep 5?

perf stat -I is in milliseconds, whereas perf record -F is in HZ (1/s), so the corresponding command to perf stat -I 1000 is perf record -F 1. In fact with our more stable event/workload, this looks better:

$ perf stat -e ref-cycles -I 1000 timeout 10s awk 'BEGIN { while(1){} }'
#           time             counts unit events
 1.000089518      2,578,694,534      ref-cycles                                                  
 2.000203872      2,579,866,250      ref-cycles                                                  
 3.000294300      2,579,857,852      ref-cycles                                                  
 4.000390273      2,579,964,842      ref-cycles                                                  
 5.000488375      2,577,955,536      ref-cycles                                                  
 6.000587028      2,577,176,316      ref-cycles                                                  
 7.000688250      2,577,334,786      ref-cycles                                                  
 8.000785388      2,577,581,500      ref-cycles                                                  
 9.000876466      2,577,511,326      ref-cycles                                                  
10.000977965      2,577,344,692      ref-cycles                                                  
10.001195845            466,674      ref-cycles    

$ perf record -e ref-cycles -F 1 timeout 10s awk 'BEGIN { while(1){} }'
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.008 MB perf.data (17 samples) ]

$ perf script -F time,period        
3369070.273722:          1 
3369070.273755:          1 
3369070.273911:       3757 
3369070.273916:    3015133 
3369070.274486:          1 
3369070.274556:          1 
3369070.274657:       1778 
3369070.274662:    2196921 
3369070.275523: 47192985748 
3369072.663696: 2578692405 
3369073.663547: 2579122382 
3369074.663609: 2580015300 
3369075.664085: 2579873741 
3369076.664433: 2578638211 
3369077.664379: 2578378119 
3369078.664175: 2578166440 
3369079.663896: 2579238122 

So you see, eventually the results are stable also for perf record -F. Unfortunately the documentation of perf record is very poor. You can learn what the settings -c and -F mean by looking at the documentation of the underlying system call man perf_event_open:

sample_period, sample_freq A "sampling" event is one that generates an overflow notification every N events, where N is given by sample_period. A sampling event has sample_period > 0. When an overflow occurs, requested data is recorded in the mmap buffer. The sample_type field controls what data is recorded on each overflow.

sample_freq can be used if you wish to use frequency rather than period. In this case, you set the freq flag. The kernel will adjust the sampling period to try and achieve the desired rate. The rate of adjustment is a timer tick.

So while perf stat uses an internal timer to read the value of the counter every -i milliseconds, perf record sets an event overflow counter to take a sample every -c events. That means it takes a sample every N events (e.g. every N page-fault or cycles). With -F, it it tries to regulate this overflow value to achieve the desired frequency. It tries different values and tunes it up/down accordingly. This eventually works for counters with a stable rate, but will get erratic results for dynamic events.

Answer from Zulan on Stack Overflow
🌐
Brendan Gregg
brendangregg.com › perf.html
Linux perf Examples
My perf-tools collection (github) uses both perf_events and ftrace as needed. This page includes my examples of perf_events. A table of contents: Key sections to start with are: Events, One-Liners, Presentations, Prerequisites, CPU statistics, Timed Profiling, and Flame Graphs.
Top answer
1 of 1
12

First of all, your test case of using sleep and page-faults is not an ideal test case. There should be no page fault events during the sleep duration, you you can't really expect anything interesting. For the sake of easier reasoning I suggest to use the ref-cycles (hardware) event and a busy workload such as awk 'BEGIN { while(1){} }'.

Question 1: It is my understanding that perf stat gets a "summary" of counts but when used with the -I option gets the counts at the specified millisecond interval. With this option does it sum up the counts over the interval or get the average over the interval, or something else entirely? I assume it is summed up.

Yes. The values are just summed up. You can confirm that by testing:

$ perf stat -e ref-cycles -I 1000 timeout 10s awk 'BEGIN { while(1){} }'
#           time             counts unit events
 1.000105072      2,563,666,664      ref-cycles                                                  
 2.000267991      2,577,462,550      ref-cycles                                                  
 3.000415395      2,577,211,936      ref-cycles                                                  
 4.000543311      2,577,240,458      ref-cycles                                                  
 5.000702131      2,577,525,002      ref-cycles                                                  
 6.000857663      2,577,156,088      ref-cycles                                                  

[ ... snip ... ]
[ Note that it may not be as nicely consistent on all systems due dynamic frequency scaling ]

$ perf stat -e ref-cycles -I 3000 timeout 10s awk 'BEGIN { while(1){} }' 
#           time             counts unit events
 3.000107921      7,736,108,718      ref-cycles                                                  
 6.000265186      7,732,065,900      ref-cycles                                                  
 9.000372029      7,728,302,192      ref-cycles     

Question 2: Why doesn't perf stat -e <event1> -I 1000 sleep 5 give about the same counts as if I summed up the counts over each second for the following command perf record -e <event1> -F 1000 sleep 5?

perf stat -I is in milliseconds, whereas perf record -F is in HZ (1/s), so the corresponding command to perf stat -I 1000 is perf record -F 1. In fact with our more stable event/workload, this looks better:

$ perf stat -e ref-cycles -I 1000 timeout 10s awk 'BEGIN { while(1){} }'
#           time             counts unit events
 1.000089518      2,578,694,534      ref-cycles                                                  
 2.000203872      2,579,866,250      ref-cycles                                                  
 3.000294300      2,579,857,852      ref-cycles                                                  
 4.000390273      2,579,964,842      ref-cycles                                                  
 5.000488375      2,577,955,536      ref-cycles                                                  
 6.000587028      2,577,176,316      ref-cycles                                                  
 7.000688250      2,577,334,786      ref-cycles                                                  
 8.000785388      2,577,581,500      ref-cycles                                                  
 9.000876466      2,577,511,326      ref-cycles                                                  
10.000977965      2,577,344,692      ref-cycles                                                  
10.001195845            466,674      ref-cycles    

$ perf record -e ref-cycles -F 1 timeout 10s awk 'BEGIN { while(1){} }'
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.008 MB perf.data (17 samples) ]

$ perf script -F time,period        
3369070.273722:          1 
3369070.273755:          1 
3369070.273911:       3757 
3369070.273916:    3015133 
3369070.274486:          1 
3369070.274556:          1 
3369070.274657:       1778 
3369070.274662:    2196921 
3369070.275523: 47192985748 
3369072.663696: 2578692405 
3369073.663547: 2579122382 
3369074.663609: 2580015300 
3369075.664085: 2579873741 
3369076.664433: 2578638211 
3369077.664379: 2578378119 
3369078.664175: 2578166440 
3369079.663896: 2579238122 

So you see, eventually the results are stable also for perf record -F. Unfortunately the documentation of perf record is very poor. You can learn what the settings -c and -F mean by looking at the documentation of the underlying system call man perf_event_open:

sample_period, sample_freq A "sampling" event is one that generates an overflow notification every N events, where N is given by sample_period. A sampling event has sample_period > 0. When an overflow occurs, requested data is recorded in the mmap buffer. The sample_type field controls what data is recorded on each overflow.

sample_freq can be used if you wish to use frequency rather than period. In this case, you set the freq flag. The kernel will adjust the sampling period to try and achieve the desired rate. The rate of adjustment is a timer tick.

So while perf stat uses an internal timer to read the value of the counter every -i milliseconds, perf record sets an event overflow counter to take a sample every -c events. That means it takes a sample every N events (e.g. every N page-fault or cycles). With -F, it it tries to regulate this overflow value to achieve the desired frequency. It tries different values and tunes it up/down accordingly. This eventually works for counters with a stable rate, but will get erratic results for dynamic events.

🌐
Perfwiki
perfwiki.github.io › main › tutorial
Introduction - perf: Linux profiling with performance counters
perf stat -B dd if=/dev/zero of=/dev/null count=1000000 1000000+0 records in 1000000+0 records out 512000000 bytes (512 MB, 488 MiB) copied, 2.14421 s, 239 MB/s Performance counter stats for 'dd if=/dev/zero of=/dev/null count=1000000': 621 context-switches # 304.3 cs/sec 7 cpu-migrations # 3.4 migrations/sec 72 page-faults # 35.3 faults/sec 2,040.80 msec task-clock # 0.9 CPUs utilized 6,169,665 cpu_core/branch-misses/ # 0.3 % branch_miss_rate (86.46%) 2,045,176,946 cpu_core/branches/ # 1002.1 M/sec (86.46%) 8,057,111,968 cpu_core/cpu-cycles/ # 3.9 GHz (86.46%) 9,568,892,085 cpu_core/instructi
🌐
Perfwiki
perfwiki.github.io
perf: Linux profiling with performance counters
We cannot provide a description for this page right now
🌐
Easyperf
easyperf.net › blog › 2018 › 08 › 26 › Basics-of-profiling-with-perf
Basics of profiling with perf. | Easyperf
Now, remember that by default we sample on cycles (equivalent to perf record -e cycles). With latest run we collected 247 samples. For simplicity let’s assume average period for all samples is 32300000 events. Based on that, the number of cycles it took to execute this workload is: 247 * 32300000 = 7978100000 cycles. If we compare this number with the number of counted cycles: $ perf stat -e cycles ./a_out Performance counter stats for './a_out': 7805574851 cycles 2,398101184 seconds time elapsed
🌐
OneUptime
oneuptime.com › home › blog › how to analyze system performance with perf on ubuntu
How to Analyze System Performance with perf on Ubuntu
January 15, 2026 - While perf stat provides summary statistics, perf record captures detailed profiling data that can be analyzed with perf report.
🌐
Johannst
johannst.github.io › notes › trace_profile › perf.html
perf - Notes
record stats for running process -o <file> .............. write output to file (default perf.data) -F <hz> ................ sampling frequency --call-graph <method> ..
🌐
GitHub
github.com › brendangregg › Misc › blob › master › perf_events › perf.md
Misc/perf_events/perf.md at master · brendangregg/Misc
# Listing all currently known events: perf list # CPU counter statistics for the entire system, for 5 seconds: perf stat -a sleep 5 # Count ext4 events for the entire system, for 10 seconds: perf stat -e 'ext4:*' -a sleep 10 # Show system calls by process, refreshing every 2 seconds: perf top -e raw_syscalls:sys_enter -ns comm # Sample CPU stack traces for the specified PID, at 99 Hertz, for 10 seconds: perf record -F 99 -p PID -g -- sleep 10 # Sample CPU stack traces for the entire system, at 99 Hertz, for 10 seconds: perf record -F 99 -ag -- sleep 10 # Sample CPUs at 49 Hertz, and show top addresses and symbols, live (no perf.data): perf top -F 49 # Show perf.data in an ncurses browser (TUI) if possible: perf report # Show perf.data file as a text report with a sample count column: perf report -n --stdi # List all raw events from perf.data: perf script
Author   brendangregg
Find elsewhere
🌐
GitHub
github.com › AmpereComputing › ampere-lts-kernel › issues › 54
Conflict result between 'perf stat' and 'perf kvm record' · Issue #54 · AmpereComputing/ampere-lts-kernel---DEPRECATED
July 12, 2021 - # # perf stat -e '{cycles:G,cycles:H}' -I 1000 -C 1 sleep 10 perf: exclude_user: 0, exclude_kernel: 0, exclude_hv: 0, exclude_host: 1, exclude_guest: 0, precise_ip: 0, exclude_idle: 0 perf: exclude_user: 0, exclude_kernel: 0, exclude_hv: 0, exclude_host: 0, exclude_guest: 1, precise_ip: 0, exclude_idle: 0 # time counts unit events 1.001039972 2511393328 cycles:G 1.001039972 491741673 cycles:H 2.002206506 2511526299 cycles:G 2.002206506 491895650 cycles:H 3.003344119 2515412775 cycles:G 3.003344119 487954160 cycles:H 4.003807089 2506767487 cycles:G 4.003807089 494567031 cycles:H
Author   AmpereComputing
🌐
GitHub
github.com › torvalds › linux › blob › master › tools › perf › Documentation › perf-list.txt
linux/tools/perf/Documentation/perf-list.txt at master · torvalds/linux
perf stat -e r20000038f -a sleep 1 · perf record -e r20000038f ... · It's also possible to use pmu syntax: · perf record -e r20000038f -a sleep 1 · perf record -e cpu/r20000038f/ ... perf record -e cpu/r0x20000038f/ ... · You should refer to the processor specific documentation for getting these ·
Author   torvalds
🌐
GitHub
github.com › torvalds › linux › blob › master › tools › perf › perf.c
linux/tools/perf/perf.c at master · torvalds/linux
* This is the main hub from which the sub-commands (perf stat, * perf top, perf record, perf report, etc.) are started. */ #include "builtin.h" #include "perf.h" · #include "util/build-id.h" #include "util/cache.h" #include "util/env.h" #include <internal/lib.h> // page_size ·
Author   torvalds
🌐
GitHub
gist.github.com › MangaD › bc385943fe0375b81c72d1c8da937ed9
Comprehensive Guide to Using `perf` for C++ Applications · GitHub
Sorting took: 45.7 ms Performance ... cache misses and branch mispredictions for better performance. perf record captures function-level performance data....
🌐
GitHub
github.com › torvalds › linux › blob › master › tools › perf › Documentation › perf-stat.txt
linux/tools/perf/Documentation/perf-stat.txt at master · torvalds/linux
December 17, 2018 - Don't print output, warnings or messages. This is useful with perf stat · record below to only write data to the perf.data file.
Author   torvalds
🌐
Iv-m
iv-m.github.io › notes › perf
perf_events notes - iv goes technical
2.6.32: perf shed, perf timechart, static tracepoints support · 2.6.33: dynamic probes – perf probe, perf diff, --filter · 2.6.34: buildid cache, perf lock, python scripting · 2.6.35: perf trace (but NOT strace), perf kvm, newt TUI · 2.6.36: perf report --sort cpu,comm · 2.6.37: – · 2.6.38: perf record --nodelay (live recording), perf stat -a -A (per-CPU stats) 2.6.39: perf top, perf evlist (which events were recorded), perf stat --filter ·
🌐
Linux Man Pages
man7.org › linux › man-pages › man1 › perf-record.1.html
perf-record(1) - Linux manual page
One is for core, the other is for atom. perf-stat: If there is no -e specified in perf stat, on hybrid platform, besides of software events, following events are created and added to event list in order. cpu_core/cycles/, cpu_atom/cycles/, cpu_core/instructions/, cpu_atom/instructions/, ...
🌐
Perfwiki
perfwiki.github.io › main › todo
Todo - perf: Linux profiling with performance counters
Implement --initial-delay, already available in 'perf stat', on 'perf trace'.
🌐
Linux Man Pages
man7.org › linux › man-pages › man1 › perf-stat.1.html
perf-stat(1) - Linux manual page
One is for core, the other is for atom. perf-stat: If there is no -e specified in perf stat, on hybrid platform, besides of software events, following events are created and added to event list in order. cpu_core/cycles/, cpu_atom/cycles/, cpu_core/instructions/, cpu_atom/instructions/, ...
🌐
GitHub
github.com › torvalds › linux › blob › master › tools › perf › Documentation › perf-record.txt
linux/tools/perf/Documentation/perf-record.txt at master · torvalds/linux
--stat:: Record per-thread event counts. Use it with 'perf report -T' to see · the values. · -d:: --data:: Record the sample virtual addresses. Implies --sample-mem-info. · --phys-data:: Record the sample physical addresses. · --data-page-size:: Record the sampled data address data page size.
Author   torvalds
🌐
GitHub
github.com › torvalds › linux › tree › master › tools › perf
linux/tools/perf at master · torvalds/linux
Linux kernel source tree. Contribute to torvalds/linux development by creating an account on GitHub.
Author   torvalds
🌐
GitHub
github.com › jean-airoldie › perf-cheat-sheet
GitHub - jean-airoldie/perf-cheat-sheet: Various helpfull perf commands. · GitHub
# Display coarse-grained stats perf stat ./benchmark # Record the program to analyze l8er. # Might require the binary to be build with the # frame pointer. e.i -fno-omit-frame-pointer perf record --call-graph dwarf ./benchmark # Analyzed the recording. # Reverse the call tree (from caller to callee).
Author   jean-airoldie