How to monitor specific variables in a running process?

Question

I would like to monitor, record, specific variables in an application but I am not sure what is the right way to approach this problem.

The application is a mix of C(5%) and C++(95%) and it is relatively big, around 5 million lines of code. The target OS is CentOS 7 and my application is always compiled with debugging flags (-g3 -ggdb) and I can add/remove flags if necessary to achieve what I am looking for. In the ideal scenario I don't want to instrument the code in the main application, mainly due to the size of the app, and/or add any overhead to my application.

I have not started implementing anything yet but I have two different approaches:

1. Write a second program that given a variable name, this variable is declared in my main program, it:

   • Parses the debug symbol of the main program
   • Finds the address of the variable we are interested in
   • Somehow manages to access the virtual address space of the main application and reads the value of the variable using the address we have found in the previous step

I am still not sure about how I can handle different data types, maybe I can somehow infer the data type by parsing the main program but I am now sure.

I understand that the OS won't let me easily access the memory of another process, but there should be a trick. What is that trick? Is there a more straight forward way of doing this? What is the overhead associated with this? Is there a way to also write into a variable?

2. Add the logic in the main application:

   • Pass the variable name over network to the application
   • The app reads and parses its own binary
   • Finds the address of the variable we are interested and we are done

The second approach is more interesting because everything is in its address space, so no trick is requuired. How ever I still have to know the type of the variable.

I am sure others have had similar problems but I don't know what this process is called. What should I search for to get more information? I understand this is a very general question, any pointers are highly appreciated.

Answer 1

well if you can use a debugger then you can employ a variety of monitoring techniques.
The example below is for a windows app being debugged by windbg.

but gdb is equally versatile and you should be able to adapt GDB to do what is being done here

calc is windows calculator it can work in the following radix hex, dec, oct and bin

it stores the radix in a global variable named g_nRadix

0:004> x /v calc!g_nRadix
pub global 00524058             0 calc!g_nRadix = <no type information>

now assume i want to monitor this variable and log when it is written to

i can write a small txt file that contains a few commands and run it on starting the app with windbg as below

the txt file that contains the debugger commands (aka script file)

C:\>cat monvar.txt
g @$exentry
ba w 4 calc!g_nRadix ".echotime;dd calc!g_nRadix l1;gc"
g
C:\>

data breakpoints are context specific so i execute until Address Of Entry Point (similar to _start function in linux app) first line g @$exentry

when i am at the the entry point i set a write data breakpoint on this variable and run the executable second and third line

i start the app inside the debugger as shown in the title of command prompt in the screens shot below

and when ever i change the radix in the gui

the debugger will print a time stamp and the content of the variable i am monitoring and keep running

Answer 2

What you are looking for is called value profiling. It's a technique that aims at logging the values of certain variables (like loop bounds) in order to extract all possible values and use them to fine tune the code for better performance.

One thing you should know, once you compile your code, debug info or not, there's no more notion of variable. And, if you use a compiler optimization flag, and you should, variables will be optimized out.

This said, your best option is either to go through your code and put prints manually (even through a parsing script); or you can go with a compiler plug-in.

If you are using GCC, I would recommend writing a plug-in guided by your own directive i.e. #pragma watch var which then instructs the compiler to instrument at the GIMPLE level with a function from a library - let's call it watch_var.so. This library's functions look like the following prototype : dump_int_var_value(FILE *fd, char *var_name, int var_value).

Only at the compiler level can you cleanly automate this process given that you'll know when the variable is declared, when it is used (read & written), and its type. The pragma directive mentioned above can come in two flavors :

1) Placed before variable declaration, var1 & var2 will be watched.

 #pragma watch var
 int var1, var2;

2) Placed at the beginning of a block, x and y will be watched in the first example. And, only i will be watched in the second one.

 #pragma watch var 
 int function1(int a, int b)
 {
     int x = a, y = b;
     return x + y;
 }

 #pragma watch var
 if (a < b)
 {
     for (int i = 0; i < n; i++)
        t[i] = a * i;
 }

From here, I guess you get the idea.

From where I stand, I see this as the only clean solution to handling variables without having to re-invent the wheel or delve into complicated tricks.