Count and frequency are two fundamental switches that tune the rate of sampling when using perf record (which does sampling internally).

Count

When you run perf record -c <number>, you are specifying the sample period (where "number" is the sample period). That is, for every "number"th occurrence of the event a sample will be recorded. The sample will be recorded when the performance counter that keeps track of the number of events has overflowed.

I am guessing you are obtaining the number of events with the help of perf report. Note that perf report will never report the actual number of events, but only an approximate. The number of events will keep changing as you keep tweaking the sample period. perf report will only read the perf.data file that perf record generates, and based on the size of the file generated, it makes an assumption of the number of samples recorded (by knowing the size of a sample recorded in memory). The actual number of events recorded is obtained by -

Number of events = Fixed Sample Period * Number of samples collected

where Fixed Sample Period is what you specified with perf record -c.

Frequency

This is the other way around to express the sampling period, that is to specify the average rate of samples per second (frequency) - which you can do using perf record -F. So perf record -F 1000 will record around 1000 samples per second and these samples will be generated when the hardware/PMU counter corresponding to the event overflows. This means that the kernel will dynamically adjust the sampling period to make sure that the sampling process adheres to the sampling frequency.

This is how the sample period gets updated dynamically.

Higher the sampling frequency, higher the number of samples collected (almost proportionately).

The variation in the sampling period can be seen by running the command -

sudo perf report -D -i perf.data | fgrep RECORD_SAMPLE

Whenever the sampling period keeps varying, the total number of events will keep incrementing with the variation in the sampling period. And when the sampling period remains fixed, the total number of events remain fixed and is obtained by the formula showed above. The total number of events will be approximate in both the cases.

ANSWER #1

Yes mostly. perf report does show you a summary of the trace collected. Samples collected by perf record are saved into a binary file called, by default, perf.data. The perf report command reads this file and generates a concise execution profile. By default, samples are sorted by functions with the most samples first. However, you can do much more detailed profiling also using this report.

ANSWER #2

You should ideally use perf script -D to get a trace of all data. The timestamp is in microseconds. Although, in kernels newer than the one you specify, with the help of a command line switch (-ns) you can display the time in nanoseconds as well. Here is the source -

Timestamp

It is quite hard to tell this without looking at what kind of "deltas" are you getting. Remember the period of collecting samples is usually tuned. There are two ways of specifying the rate at which to collect samples --

You can use the perf record (--c for count) to specify the period at which to collect samples. This will mean that for every c occurrences of the event that you are measuring, you will have a sample for that. You can then modify the sampling period and test various values. This means that at every two occurences of the event for which you are measuring, the counter will overflow and you will record a sample.

The other way around to express the sampling period, is to specify the average rate of samples per second (frequency) - which you can do using perf record -F. So perf record -F 1000 will record around 1000 samples per second and these samples will be generated when the hardware/PMU counter corresponding to the event overflows. This means that the kernel will dynamically adjust the sampling period. And you will get sample times at different random moments.

You can see for yourself in code here:

How perf dynamically updates time

ANSWER #3

Why not ? Ideally you should get the number of event samples collected if you do a perf report and just do a deeper analysis. Also when you do a perf record and finish recording samples, you would get a notification on the command line about the number of samples collected corresponding to the event you measured. (This may not be available in the kernel module you use, I would suggest you switch to a newer linux version if possible!). The number of samples should be the raw count - not the period.

If your period is 100 - it means that for the whole duration of the trace, perf recorded every 100th event. That means, if a total of 1000 events happened for the trace duration, perf approximately collected event 1, 100, 200, 300...1000.

Yes the samples recorded are not only from the application. In fact, you can use switches like this : perf record -e <event-name:u> or <event-name:k> (u for userspace and k for kernel) to record events. Additionally perf records samples from shared libraries as well. (Please consult the perf man-page for more details).

As I said previously, perf report should be an ideal tool to calculate the number of samples of event cycles recorded by perf. The number of events collected/recorded is not exact because it is simply not possible for hardware to record all cycle events. This is because recording and preparing details of all the events require the kernel to maintain a ring buffer which gets written to periodically as and when the counter overflows. This writing to the buffer happens via interrupts. They take up a fraction of CPU time- this time is lost and could have been used to record events which will now be lost as the CPU was busy servicing interrupts. You can get a really great estimate by perf even then, though.

CONCLUSION

perf does especially what it intends to do given the limitations of hardware resources we have at hand currently. I would suggest going through the man-pages for each command to understand better.

QUESTIONS

2. I assume you are looking at perf report. I also assume you are talking about the overhead % in perf report. Theoretically, it can be considered to be an arrangement of data from the highest to least occurrence as you specified. But, there are many underlying details that you need to consider and understand to properly make sense of the output. It represents which function has the most overhead (in terms of the number of events that occurred in that function ). There is also a parent-child relationship, based on which function calls which function, between all the functions and their overheads. Please use the Perf Report link to understand more.

3. As you know already events are being sampled, not counted. So you cannot accurately get the number of events, but you will get the number of samples and based on the tuned frequency of collecting samples, you will also get the raw count of the number of events ( Everything should be available to you with the perf report output ).

There is perf stat option "interval-print" of -I N where N is millisecond interval to do interval counter printing every N milliseconds (N>=10): http://man7.org/linux/man-pages/man1/perf-stat.1.html

  -I msecs, --interval-print msecs
       Print count deltas every N milliseconds (minimum: 10ms) The
       overhead percentage could be high in some cases, for instance
       with small, sub 100ms intervals. Use with caution. example: perf
       stat -I 1000 -e cycles -a sleep 5

  For best results it is usually a good idea to use it with interval
   mode like -I 1000, as the bottleneck of workloads can change often.

There is also importing results in machine-readable form, and with -I first field is datetime:

With -x, perf stat is able to output a not-quite-CSV format output ... optional usec time stamp in fractions of second (with -I xxx)

vmstat, systat-family tools iostat, mpstat, etc periodic printing is -I 1000 of perf stat (every second), for example system-wide (add -A to separate cpu counters):

  perf stat -a -I 1000

The option is implemented in builtin-stat.c http://lxr.free-electrons.com/source/tools/perf/builtin-stat.c?v=4.8 __run_perf_stat function

531 static int __run_perf_stat(int argc, const char **argv)
532 {
533         int interval = stat_config.interval;

For perf stat -I 1000 with some program argument (forks=1), for example perf stat -I 1000 sleep 10 there is interval loop (ts is the millisecond interval converted to struct timespec):

639                 enable_counters();
641                 if (interval) {
642                         while (!waitpid(child_pid, &status, WNOHANG)) {
643                                 nanosleep(&ts, NULL);
644                                 process_interval();
645                         }
646                 }
666         disable_counters();

For variant of system-wide hardware performance monitor counting and forks=0 there is other interval loop

658                 enable_counters();
659                 while (!done) {
660                         nanosleep(&ts, NULL);
661                         if (interval)
662                                 process_interval();
663                 }
666         disable_counters();

process_interval() http://lxr.free-electrons.com/source/tools/perf/builtin-stat.c?v=4.8#L347 from the same file uses read_counters(); which loops over event list and invokes read_counter() which loops over all known threads and all cpus and starts actual reading function:

306         for (thread = 0; thread < nthreads; thread++) {
307                 for (cpu = 0; cpu < ncpus; cpu++) {
...
310                         count = perf_counts(counter->counts, cpu, thread);
311                         if (perf_evsel__read(counter, cpu, thread, count))
312                                 return -1;

perf_evsel__read is the real counter read while program is still running:

1207 int perf_evsel__read(struct perf_evsel *evsel, int cpu, int thread,
1208                      struct perf_counts_values *count)
1209 {
1210         memset(count, 0, sizeof(*count));
1211 
1212         if (FD(evsel, cpu, thread) < 0)
1213                 return -EINVAL;
1214 
1215         if (readn(FD(evsel, cpu, thread), count, sizeof(*count)) < 0)
1216                 return -errno;
1217 
1218         return 0;
1219 }