Make sure you compiled the code with -fno-omit-frame-pointer gcc option.

Answer from Andriy on Stack Overflow
🌐
Brendan Gregg
brendangregg.com › perf.html
Linux perf Examples
Older versions of perf (or if you use --stdio in the new version) print the call graph as a tree, annotated with percentages: # perf report --stdio # ======== # captured on: Mon Jan 26 07:26:40 2014 # hostname : dev2 # os release : 3.8.6-ubuntu-12-opt # perf version : 3.8.6 # arch : x86_64 # nrcpus online : 8 # nrcpus avail : 8 # cpudesc : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz # cpuid : GenuineIntel,6,44,2 # total memory : 8182008 kB # cmdline : /usr/bin/perf record -F 99 -a -g -- sleep 30 # event : name = cpu-clock, type = 1, config = 0x0, config1 = 0x0, config2 = ...
🌐
Linux Man Pages
man7.org › linux › man-pages › man1 › perf-record.1.html
perf-record(1) - Linux manual page
It will produce call graphs from the hardware LBR registers. The main limitation is that it is only available on new Intel platforms, such as Haswell. It can only get user call chain. It doesn't work with branch stack sampling at the same time. When "dwarf" recording is used, perf also records ...
🌐
Linux Man Pages
man7.org › linux › man-pages › man1 › perf-report.1.html
perf-report(1) - Linux manual page
To generate meaningful output, ... perf record -d -W and using a special event -e cpu/mem-loads/p or -e cpu/mem-stores/p. See perf mem for simpler access. --percent-limit Do not show entries which have an overhead under that percent. (Default: 0). Note that this option also sets the percent limit (threshold) of callchains. However the default value of callchain threshold is different than the default value of hist entries. Please see the --call-graph option for ...
🌐
DBI Services
dbi-services.com › accueil › linux perf-top basics: understand the %
Linux perf-top basics: understand the %
March 14, 2021 - sudo perf record -F99 -g --call-graph fp --delay=5 -v -p $(pgrep -d, yb-tserver) -a sleep 10 sudo perf report --call-graph ,,,,callee --symbol-filter=RawUncompress
🌐
GitHub
github.com › llvm › llvm-project › issues › 53156
perf record --call-graph dwarf does not support ld.lld's default --rosegment -z noseparate-code layout · Issue #53156 · llvm/llvm-project
January 12, 2022 - $ cat foo.c #include <stdio.h> void foo(int d) { int s = 0; for (int i = 0; i < 1000000; i++) { printf(""); s += i / d; } } int main() { foo(100000); } $ perf --version perf version 5.10.89 $ clang -v foo.c && perf record --call-graph dwarf ./a.out && perf report | head -n 30 clang -v foo.c && perf record --call-graph dwarf ./a.out && perf report | head -n 30 clang version 13.0.0 Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /nix/store/ry469fxcshgi8k6mavw5623ps4wyc2dp-clang-13.0.0/bin Found candidate GCC installation: /nix/store/mrqrvina0lfgrvdzfyri7sw9vxy6pyms-gcc-10.3.0/
Author   llvm
Find elsewhere
🌐
PostgreSQL Wiki
wiki.postgresql.org › wiki › Profiling_with_perf
Profiling with perf - PostgreSQL wiki
July 24, 2018 - Newer perf releases support capturing the whole user-space call stack, not just the top function on the stack. This lets it provide extremely useful call graphs. Use the --call-graph dwarf option to perf record to enable this, and the -g option to perf report to display the results.
Top answer
1 of 1
4

Man page of pref report documents the call chains display with children accumulation:

  --children
       Accumulate callchain of children to parent entry so that then can
       show up in the output. The output will have a new "Children"
       column and will be sorted on the data. It requires callchains are
       recorded. See the ‘overhead calculation’ section for more
       details. Enabled by default, disable with --no-children.

I can recommend you to try non-default mode with --no-children option of perf report (or perf top -g --no-children -p $PID_OF_PROGRAM)

So in default mode when there is some callchain data in perf.data file, perf report will calculate "self" and "self+children" overhead and sort on accumulated data. It means that if some function f1() has 10% of "self" samples and calls some leaf function f2() with 20% of "self" samples, then f1() self+children will be 30%. Accumulated data is for all stacks where current function was mentioned: for the work done in it itself, and work in all direct and indirect children (descendants).

You can specify some of call stack sampling method in --call-graph option (dwarf / lbr / fp), and they may have some limitations. Sometimes methods (especially fp) may fail to extract parts of call stack. -fno-omit-frame-pointer option may help, but when it is used in your executable but not in some library with callback, then call stack will be extracted partially. Some very long call chains may be not extracted too by some methods. Or perf report may fail to handle some cases.

To check for truncated call chain samples, use perf script|less somewhere in the middle. In this mode it does print every recorded sample with all detected function names, check for samples not ending with main and __libc_start_main - they are truncated.

otherwise the main function would have its children column not far from 100%

Yes, for single threaded program and correctly recorded and processed call stacks, main should have something like 99% in "Children" column. For multithreaded programs second and other threads will have another root node like start_thread.

🌐
Mark Hansen's Blog
markhansen.co.nz › profiler-uis
Linux perf Profiler UIs - Mark Hansen's Blog
October 7, 2021 - perf can interrupt threads to record thread's stack traces, triggered by an event (e.g. a thread context switch, or a syscall) or on a regular schedule (e.g. 4000 times a second), e.g.: # Prerequisite: install debug symbols.
🌐
Trophy
trofi.github.io › posts › 215-perf-and-dwarf-and-fork.html
perf and DWARF and fork()
April 10, 2020 - Luckily basic debugging information has a way to encode equivalent information using DWARF format. perf knows how to unwind DWARF using call stack snapshots with --call-graph=dwarf: $ perf record -F 99 -g --call-graph=dwarf -- \ /usr/lib/llvm/10/bin/clang -w -x c++ -std=c++1z -O0 -fstrict-aliasing -c A.cpp.c -o A.o $ perf report Children Self Command Shared Object Symbol + 11,12% 0,00% clang libclang-cpp.so.10 [.] (anonymous namespace)::EmitAssemblyHelper::EmitAssembly + 11,03% 0,00% clang libLLVM-10.so [.] llvm::legacy::PassManagerImpl::run + 9,70% 0,09% clang libLLVM-10.so [.] llvm::FPPassMa
🌐
Ubuntu
manpages.ubuntu.com › manpages › bionic › man1 › perf-report.1.html
Ubuntu Manpage: perf-report - Read perf.data (created by perf record) and display the profile
To generate meaningful output, ... using perf record -d -W and using a special event -e cpu/mem-loads/ or -e cpu/mem-stores/. See ... mem for simpler access. --percent-limit Do not show entries which have an overhead under that percent. (Default: 0). Note that this option also sets the percent limit (threshold) of callchains. However the default value of callchain threshold is different than the default value of hist entries. Please see the --call-graph option for ...
🌐
GitHub
gist.github.com › trevnorris › f0907b010c9d5e24ea97
Here's a rundown of everything I use to do performance analysis in Node. · GitHub
$ perf probe -x ./node -a uv_write Added new events: probe_node:uv_write (on uv_write in ./node/out/Release/node) You can now use it in all perf tools, such as: perf record -e probe_node:uv_write -aR sleep 1 · Now we can create a call graph for all calls to that probe:
🌐
Gerryyang
blog.gerryyang.com › linux performance › 2021 › 06 › 21 › perf-in-action.html
Perf in Action | Gerry’s blog
June 21, 2021 - perf record -h --call-graph <record_mode[,record_size]> setup and enables call-graph (stack chain/backtrace): record_mode: call graph recording mode (fp|dwarf|lbr) record_size: if record_mode is 'dwarf', max size of stack recording (<bytes>) default: 8192 (bytes) Default: fp
🌐
DEV Community
dev.to › franckpachot › linux-perf-top-basics-understand-the-316l
Linux perf-top basics: understand the % - DEV Community
March 21, 2021 - sudo perf record -F99 -g --call-graph fp --delay=5 -v -p $(pgrep -d, yb-tserver) -a sleep 10 sudo perf report --call-graph ,,,,callee --symbol-filter=RawUncompress
🌐
Fishilico
fishilico.github.io › generic-config › sysadmin › perf-linux.html
Using perf on Linux — Generic Config
perf record $COMMAND # --branch-any: enable taken branch stack sampling # --call-graph=dwarf: enable call-graph (stack chain/backtrace) recording with DWARF information perf record --branch-any --call-graph=dwarf $COMMAND # Record a running process during 30 seconds # -a = --all-cpus: system-wide collection from all CPUs # -g (like --call-graph=fp): enable call-graph (stack chain/backtrace) recording # -p = --pid: record events on existing process ID (comma separated list) timeout 30s perf record -a -g -p $(pidof $MYPROCESS)
🌐
Ceph Documentation
docs.ceph.com › en › latest › dev › perf
Using perf — Ceph Documentation
To capture some data with call graphs: sudo perf record -p `pidof ceph-osd` -F 99 --call-graph dwarf -- sleep 60 · To view by caller (where you can see what each top function calls): sudo perf report --call-graph caller · To view by callee (where you can see who calls each top function): ...
🌐
Arch Linux Man Pages
man.archlinux.org › man › perf-record.1.en
perf-record(1) — Arch manual pages
It will produce call graphs from the hardware LBR registers. The main limitation is that it is only available on new Intel platforms, such as Haswell. It can only get user call chain. It doesn't work with branch stack sampling at the same time. When "dwarf" recording is used, perf also records ...