Profiling with OProfile

More Information

OProfile Home Page

Installation

Install using yum:

sudo yum -y install oprofile

Start the profiling daemon

Wait until you are ready to run your target program before you do this.

I think you need to run these commands as root.

"--reset" cleans out old data.

"--no-vmlinux" expresses that you are too lazy to find a kernel in the right format (w/ symbols, uncompressed)

sudo opcontrol --reset
sudo opcontrol --no-vmlinux
sudo opcontrol --start

The following command was used by someone else to get kernel symbols:

sudo opcontrol --start \
               --vmlinux=/usr/lib/debug/lib/modules/`uname -r`/vmlinux \
               --separate=kernel

kernel-debuginfo might be a package you need.

Run your target program

Longer and harder is better.

Dump the profile data

You don't have to stop your program to dump the data.

You don't even have to stop profiling, but it will keep the results static for this HOWTO.

I think these commands need to be run as root too.

sudo opcontrol --dump
sudo opcontrol --shutdown

Generate a basic system-wide report

We'll add the "-f" flag so we get long filenames in the report.

sudo opreport -f

We get output that looks like this:

CPU: PIII, speed 800 MHz (estimated)
Counted CPU_CLK_UNHALTED events (clocks processor is not halted) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
   861982 29.6616 /lib/libc-2.4.so
   716161 24.6437 /usr/lib/libstdc++.so.6.0.8
   504902 17.3741 /usr/lib/libbz2.so.1.0.3
   461460 15.8792 /home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so
   101235  3.4836 /no-vmlinux
    99956  3.4396 /home/ksedgwic/skyler/tam.trunk/modules/MemDataWindow/Linux.DBGOBJ/RTQE-MDW.so
    73765  2.5383 /usr/lib/libboost_filesystem.so.1.33.1
    15907  0.5474 /home/ksedgwic/skyler/tam.trunk/channel/src/Linux.DBGOBJ/librtqe-channel.so
    11904  0.4096 /usr/bin/oprofiled

Detailed listing

Run the opreport again, we add: * '-l' to list all symbols * '--demangle=smart' to demangle C++ symbols * '--debug-info' to list source code locations

sudo opreport -l --demangle=smart --debug-info

Detailed listing for a specific application or object

Run the opreport again, we add: * '-l' to list all symbols * '--demangle=smart' to demangle C++ symbols * '--debug-info' to list source code locations

We also add the full path of the object of interest

# Kernel
sudo opreport -l /usr/lib/debug/lib/modules/`uname -r`/vmlinux

# User module
sudo opreport -l --demangle=smart --debug-info \
    /home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so

We get the following output:

samples  %        symbol name
32489     7.0405  MDL::FileLoader::readNext()
30441     6.5967  MDL::Field::Field(MDL::Field const&)
29384     6.3676  MDL::FileLoader::getMsgFromFile(string const&)
24771     5.3680  MDL::Field::~Field()
19645     4.2571  __i686.get_pc_thunk.bx
11751     2.5465  _Rb_tree<string, pair<string const, MDL::Field>, string const>
::insert_equal(pair<string const, MDL::Field> const&)
9908      2.1471  MDL::Allocator<double>::~Allocator()
9600      2.0804  vector<MDL::Field>::_M_insert_aux(vector<MDL::Field>::iterator
, MDL::Field const&)
9147      1.9822  MDL::MessageImpl::addField(MDL::Field const&)
9057      1.9627  MDL::Allocator<string>::~Allocator()
8985      1.9471  .plt
7595      1.6459  MDL::Allocator<double>::copy() const
7448      1.6140  MDL::IAlloc::~IAlloc()
...

Call graph listing for specific application or object

Similar command to the previous section, but use "-c" for call graph output.

sudo opreport -c --demangle=smart \
     /home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so

We get the following output:

samples  %        symbol name
-------------------------------------------------------------------------------
32489     7.0405  MDL::FileLoader::readNext()
  32489    100.000  MDL::FileLoader::readNext() [self]
-------------------------------------------------------------------------------
30441     6.5967  MDL::Field::Field(MDL::Field const&)
  30441    100.000  MDL::Field::Field(MDL::Field const&) [self]
-------------------------------------------------------------------------------
29384     6.3676  MDL::FileLoader::getMsgFromFile(string const&)
  29384    100.000  MDL::FileLoader::getMsgFromFile(string const&) [self]
-------------------------------------------------------------------------------
24771     5.3680  MDL::Field::~Field()
  24771    100.000  MDL::Field::~Field() [self]
-------------------------------------------------------------------------------
19645     4.2571  __i686.get_pc_thunk.bx
  19645    100.000  __i686.get_pc_thunk.bx [self]
-------------------------------------------------------------------------------
11751     2.5465  _Rb_tree<string, pair<string const, MDL::Field>, string const>::insert_equal(pair<string const, MDL::Field> const&)
  11751    100.000  _Rb_tree<string, pair<string const, MDL::Field>, string const>::insert_equal(pair<string const, MDL::Field> const&) [self]
-------------------------------------------------------------------------------
9908      2.1471  MDL::Allocator<double>::~Allocator()
  9908     100.000  MDL::Allocator<double>::~Allocator() [self]
-------------------------------------------------------------------------------
9600      2.0804  vector<MDL::Field>::_M_insert_aux(vector<MDL::Field>::iterator, MDL::Field const&)
  9600     100.000  vector<MDL::Field>::_M_insert_aux(vector<MDL::Field>::iterator, MDL::Field const&) [self]
-------------------------------------------------------------------------------
9147      1.9822  MDL::MessageImpl::addField(MDL::Field const&)
  9147     100.000  MDL::MessageImpl::addField(MDL::Field const&) [self]
-------------------------------------------------------------------------------

This output is a little suspicious (why are all the callee vectors 100%)?

Annotating source code

sudo opannotate --source --output-dir=/tmp/annotated \
     /home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so

Excerpt from /tmp/annotated/home/ksedgwic/skyler/tam.trunk/modules/MDLoader/FileLoader.cpp:

               :                // remove trailing newline character
 24703  5.3532 :                size_t length = strlen(line) ;
  1246  0.2700 :                if (line[length-1] == '\n')
               :                {
   593  0.1285 :                        line[--length] = '\0' ;
               :                }
   516  0.1118 :                sline = line ;
               :
  1152  0.2496 :                switch (state)
               :                {
               :                case MSBegin:
               :                        // read info, change to MSBody state
    71  0.0154 :                        m_msgheadline = sline ;
    20  0.0043 :                        state = MSBody ;
     1 2.2e-04 :                        break ;
               :                case MSBody:
               :                        // if blank line, save and
               :                        // change to MSComplete state to terminate loop
  1160  0.2514 :                        if (sline.length() == 0 || sline.find("\n") == 0)
               :                        {
    14  0.0030 :                                state = MSComplete ;
               :                                break ;
               :                        }
               :
  1432  0.3103 :                        s += sline + "\n" ;
               :                        break ;
               :                default:
               :                        abort() ;
               :                }
               :        }
               :

Annotating assembly code

This is perhaps the most useful mode of all. All of the output appears in one file of interleaved source and assembly.

The file is sorted with the most significant routines at the top.

Best of all STL classes and other inlined code are placed where they are used, and fully accounted.

sudo opannotate --source --assembly \
     /home/ksedgwic/skyler/tam.trunk/modules/MDLoader/Linux.DBGOBJ/RTQE-MDLoader.so \
     > /tmp/annotated/assem