Multiple cgroups: perf trace -G A -e sched:*switch -G B the syscall ones go to the 'A' cgroup, the sched:sched_switch goes to the 'B' cgroup. --filter-pids= Filter out events for these pids and for trace itself (comma separated list). -v, --verbose Increase the verbosity level.

The CPU process summarizes the scheduled work for each CPU core, and also has a special track showing any threads blocked for more than 3ms. (You can change this threshold via command-line options to perf_trace_viewer.)

A perf record command was used to trace the block:block_rq_issue probe, which fires when a block device I/O request is issued (disk I/O). Options included -a to trace all CPUs, and -g to capture call graphs (stack traces). Trace data is written to a perf.data file, and tracing ended when Ctrl-C ...

Trace the first 4 open, openat or open_by_handle_at syscalls (in the future more syscalls may match here): $ perf trace -e open* --max-events 4 [root@jouet perf]# trace -e open* --max-events 4 2272.992 ( 0.037 ms): gnome-shell/1370 openat(dfd: CWD, filename: /proc/self/stat) = 31 2277.481 ( 0.139 ms): gnome-shell/3039 openat(dfd: CWD, filename: /proc/self/stat) = 65 3026.398 ( 0.076 ms): gnome-shell/3039 openat(dfd: CWD, filename: /proc/self/stat) = 65 4294.665 ( 0.015 ms): sed/15879 openat(dfd: CWD, filename: /etc/ld.so.cache, flags: CLOEXEC) = 3 $

July 4, 2022 - Short tutorial about perf trace, an strace-inspired system troubleshooting functionality in the Linux perf toolbox, including basic usage, flags and options, some examples, and more

The perf ftrace command provides a collection of subcommands which use kernel’s ftrace infrastructure. 'perf ftrace trace' is a simple wrapper of the ftrace. It only supports single thread tracing currently and just reads trace_pipe in text and then write it to stdout.

It is available under Perf sampling. In order to have the symbol resolved to their names, the symbols in libraries should be available if the mmap[2] events are in the trace (the --all argument should have been used when converting). To resolve the kernel symbols, the mapping file should be added to the trace.

Performance counters are CPU hardware registers that count hardware events such as instructions executed, cache-misses suffered, or branches mispredicted. They form a basis for profiling applications to trace dynamic control flow and identify hotspots. perf provides rich generalized abstractions over hardware specific capabilities.

December 18, 2018 - # perf trace -e sched:*switch/nr=2/,block:*_plug/nr=4/,block:*_unplug/nr=1/,net:*dev_queue/nr=3,max-stack=16/

Perf is able to locally access CoreSight trace data and store it to the output perf data files. This data can then be later decoded to give the instructions that were traced for debugging or profiling purposes.

When tracing pagefaults, the format of the trace is as follows: <min|maj>fault [<ip.symbol>+<ip.offset>] ⇒ <m[blue]addr.dso [at] addr.offsetm[] ... • ip.symbol shows symbol for instruction pointer (the code that generated the fault); if no debug symbols available, perf trace will print raw IP;

Trace the first 4 open, openat or open_by_handle_at syscalls (in the future more syscalls may match here): $ perf trace -e open* --max-events 4 [root@jouet perf]# trace -e open* --max-events 4 2272.992 ( 0.037 ms): gnome-shell/1370 openat(dfd: CWD, filename: /proc/self/stat) = 31 2277.481 ( 0.139 ms): gnome-shell/3039 openat(dfd: CWD, filename: /proc/self/stat) = 65 3026.398 ( 0.076 ms): gnome-shell/3039 openat(dfd: CWD, filename: /proc/self/stat) = 65 4294.665 ( 0.015 ms): sed/15879 openat(dfd: CWD, filename: /etc/ld.so.cache, flags: CLOEXEC) = 3 $

HPerf reads a Linux perf trace (perf.data) and annotates the corresponding disassembly.

May 27, 2025 - Linux perf is a powerful performance analysis tool that uses hardware performance counters and kernel tracepoints to profile applications and the system.

The 'perf' tool is the profiling and tracing tool that comes bundled with the Linux kernel. Don't let the fact that it's part of the kernel fool you into thinking that it's only for tracing and profiling the kernel - you can indeed use it to trace and profile just the kernel , but you can also ...

July 29, 2021 - Last night while I was working on my POUG 2021 presentation I looked up what’s new with perf(1) since I last spoke about profiling and tracing Linux on-premises and in the cloud back in 2017. Unsurprisingly, quite a few things have changed. As with every tracer/profiler, using tools such as perf trace(1) and strace(1) outside of lab/playground environments is discouraged as it can unexpectedly change a process’s behaviour, slow it down or even crash the process under investigation This article is for educational purposes only and purely to be used for research on a lab VM where potential damage is limited to the individual experimenting with the tools.

We therefore present a solution to gather performance data using existing Linux performance monitoring interfaces. A combination of sampling and careful instrumentation using either perf or ftrace allows us to obtain detailed performance traces with manageable overhead.

In Brendan Gregg's page on Linux tracers, he says “[perf_events] can do many things, but if I had to recommend you learn just one [tool], it would be CPU profiling”. Tracing tools are cumbersome enough that his top recommendation on his page about tracing tools is to learn a profiling tool!