Nested Structures in IDA

Question

I'm trying to figure out how to explore structs in IDA, and getting stuck on the basics. I wrote a very basic test app;

int main() {
    const int kBits = 1024;
    const int kExp = 3;

    RSA *rsa = RSA_generate_key(kBits, kExp, 0, 0);
    RSA_print_fp(stdout, rsa, 0);

    RSA_free(rsa);
    return 0;
}

Then I imported the OpenSSL headers, so IDA now understands what RSA* and BIGNUM* are.

Then, dropped a breakpoint in the pseudocode view;

Now, if I hover over v3, I see it's parsing the struct. So I want to do something like grab the value of rsa->d->dmax, but there doesn't seem to be a way to do this? I can't seem to find a way to access v3 by name in IDAPython, nor can I seem to 'walk' the structure to get to the int that lives at dmax. Is there some way to do this?

Answer 1

Here are the steps:

1. Set breakpoint. Run the local debugger with F9. Open Debugger dropdown menu form menu bar and choose "Take memory snapshot". Here is the screenshot:

2. Double click on the variables that you want to know (here the RSA* pointer). IDA will take you to the stack view (aka. IDA View-RIP). Do not move the cursor otherwise you will get different value. Press N to name the variable in that stack view. "Rename address" window will pop-up. Here is the screenshot:

3. After naming the variable in that stack view, press Y to add the data type i.e. structure type. For this case it will be RSA, not the pointer because all stack variables placed linearly. Here is the screenshot:

Now you can see all the struct members value.

Answer 2

My knowledge of the IDApython API isn't the most complete, but what I would do is

1. Read the source (or look in IDA) to find the location (byte offsets) of d in rsa_st and dmax in bignum_st. This is easy since the members of the structures above d and dmax are pointers or int types.
2. Inspect the disassembly just before the call and determine the location of v3 - from the pseudocode it appears to be stored in the stack. If you are lucky, pressing tab while your cursor is on v3 will take you to some mov operation in disassembly.
3. When the breakpoint is hit, use a combination of idc.get_reg_value and idc.get_bytes commands (see https://www.hex-rays.com/products/ida/support/idadoc/162.shtml), using the result from step 2 as a starting point.