-T, --trace-funcs= Set the function name to get the histogram. Unlike perf ftrace trace, it only allows single function to calculate the histogram. -e, --events= Set the pair of events to get the histogram. The histogram is calculated by the time difference between the two events from the same ...

April 23, 2013 - Example below: # perf ftrace record sleep 0.1 # ls -l perf.data.dir total 5568 -rw-r--r--. 1 root root 3514375 Apr 23 16:43 perf.header -rw-r--r--. 1 root root 90112 Apr 23 16:43 trace-cpu0.buf -rw-r--r--. 1 root root 0 Apr 23 16:43 trace-cpu1.buf -rw-r--r--. 1 root root 2093056 Apr 23 16:43 trace-cpu2.buf -rw-r--r--. 1 root root 0 Apr 23 16:43 trace-cpu3.buf # perf ftrace show overriding event (11) ftrace:funcgraph_entry with new print handler overriding event (10) ftrace:funcgraph_exit with new print handler sleep-11573 [000] 717758.728678862: funcgraph_entry: finish_task_switch() { sleep-11

# perf list | awk -F: '/Tracepoint event/ { lib[$1]++ } END { for (l in lib) { printf " %-16.16s %d\n", l, lib[l] } }' | sort | column alarmtimer 4 i2c 8 page_isolation 1 swiotlb 1 block 19 iommu 7 pagemap 2 syscalls 614 btrfs 51 irq 5 power 22 task 2 cgroup 9 irq_vectors 22 printk 1 thermal ...

This requires CONFIG_FTRACE and other FTRACE options depending on the tool. Some tools (eg, funccount) require CONFIG_FUNCTION_PROFILER. Requires the "perf" command to be installed. This is in the linux-tools-common package. After installing that, perf may tell you to install an additional linux-tools package (linux-tools-kernel_version).

Example below: # perf ftrace record sleep 0.1 # ls -l perf.data.dir total 5568 -rw-r--r--. 1 root root 3514375 Apr 23 16:43 perf.header -rw-r--r--. 1 root root 90112 Apr 23 16:43 trace-cpu0.buf -rw-r--r--. 1 root root 0 Apr 23 16:43 trace-cpu1.buf -rw-r--r--. 1 root root 2093056 Apr 23 16:43 trace-cpu2.buf -rw-r--r--. 1 root root 0 Apr 23 16:43 trace-cpu3.buf # perf ftrace show overriding event (11) ftrace:funcgraph_entry with new print handler overriding event (10) ftrace:funcgraph_exit with new print handler 0) 0.065 us | __fsnotify_parent(); 0) | fsnotify() { 0) 0.060 us | __srcu_read_lock(

March 18, 2023 - Here’s an example of a simple ftrace command that traces all syscalls made by a specific process: echo syscalls > /sys/kernel/debug/tracing/current_tracer echo $$ > /sys/kernel/debug/tracing/set_ftrace_pid cat /sys/kernel/debug/tracing/trace ...

# perf ftrace -T __kmalloc ./lock 20) + 17.698 us | __kmalloc(); 20) 3.167 us | __kmalloc(); 20) 0.952 us | __kmalloc();

Multiple CPUs can be provided as a comma separated list with no space like: 0,1. Ranges of CPUs are specified with -: 0-2. Default is to trace on all online CPUs. -T, --trace-funcs= Only trace functions given by the argument. Multiple functions can be given by using this option more than once. The function argument also can be a glob pattern. It will be passed to set_ftrace_filter in tracefs.

ftrace command is a simple wrapper of kernel’s ftrace functionality. It only supports single thread tracing currently and just reads trace_pipe in text and then write it to stdout. The following options apply to perf ftrace.

Only trace functions given by the argument. Multiple functions can be given by using this option more than once. The function argument also can be a glob pattern. It will be passed to set_ftrace_filter in tracefs.

You can put arbitrary functionality into that callback. Correct me if I'm wrong but since the trace_mce_record() is a tracepoint, we don't want to register callbacks over the perf_event* interface but rather use the ftrace path, like Steven's example with register_trace_sched_switch().

'perf ftrace latency' calculates execution latency of a given function (optionally with BPF) and display it as a histogram.

July 31, 2024 - $ sudo perf ftrace profile -G __x64_sys_perf_event_open -- \ perf stat -e cycles -C1 true 2> /dev/null | head # Total (us) Avg (us) Max (us) Count Function 65.611 65.611 65.611 1 __x64_sys_perf_event_open 30.527 30.527 30.527 1 anon_inode_getfile 30.260 30.260 30.260 1 __anon_inode_getfile 29.700 29.700 29.700 1 alloc_file_pseudo 17.578 17.578 17.578 1 d_alloc_pseudo 17.382 17.382 17.382 1 __d_alloc 16.738 16.738 16.738 1 kmem_cache_alloc_lru 15.686 15.686 15.686 1 perf_event_alloc 14.012 7.006 11.264 2 obj_cgroup_charge # Reviewed-by: Ian Rogers <irogers@google.com> Signed-off-by: Namhyung Ki

Brendan Gregg (our linux debugging tools hero) has repository of tools that use ftrace to give you information about various things like IO latency. They’re all in his perf-tools repository on GitHub.

'perf ftrace latency' calculates execution latency of a given function (optionally with BPF) and display it as a histogram.

static bool check_ftrace_capable(void) { bool used_root; · if (perf_cap__capable(CAP_PERFMON, &used_root)) return true; · if (!used_root && perf_cap__capable(CAP_SYS_ADMIN, &used_root)) return true; · pr_err("ftrace only works for %s!\n", used_root ?

ftrace: The hidden light switch (lwn.net), perf-tools (github), ftrace.txt ... Then there's also the add on tracers, like Systemtap, LTTng, sysdig, etc, which I didn't cover in 15 minutes. My 15 minute tracing demo was inspired by Greg Law's excellent cppcon talk Give me 15 minutes & I'll change your view of GDB. Since then, I've also written about GDB here, with a full GDB example (tutorial).

March 18, 2019 - Only trace functions given by the argument. Multiple functions can be given by using this option more than once. The function argument also can be a glob pattern. It will be passed to set_ftrace_filter in tracefs.

'perf ftrace latency' calculates execution latency of a given function (optionally with BPF) and display it as a histogram.

Performance analysis tools based on Linux perf_events (aka perf) and ftrace - perf-tools/examples/reset-ftrace_example.txt at master · brendangregg/perf-tools