Finding the encryption key in a binary - IDA Pro and how an S-box might be represented in the binary

Question

In practicing reading source code and analysing I've been looking at a program called aescrypt2_huawei which floats around the web and encrypts/decrypts the XML config file from Huawei routers like the HG8245 router installed by Internet Service Providers. The router is discussed on a number of blogs and the aescrypt2 can be downloaded and examined. I have also found the source code and put it on GitHub.

So far I've found:

• NIST SHA256 Test Vector in the source
• inflate 1.2.5 calling code
• Identified the code path to the decrypt section

What I haven't found:

• Any S-boxes for AES
• The key

Running findcrypt in IDA Pro is also not turning up anything which leads me to believe my findcrypt plugin isn't working. However, I then start looking for the S-boxes sequence 0x63 0x7c which can easily be found in the source code and IDA pro doesn't find that either... I then dropped to hexdump and searched for 0x63 and it is not there. Given that I can see the S-box in the source and I just compiled the binary myself what am I doing wrong?

Shouldn't the forward S-box appear in the data segment more or less as-is?

And now that I've found the key through the debugger in the .bss segment why isn't the string in the data segment?

Answer 1

Initialization of the forward S-box

As you probably saw in the source-code, the forward S-box is present several times and initialized at two different places.

The first time that the forward S-box (Fsb) is declared is at the top of aes.c:

uint32 FSb[256];

This array is then dynamically generated by aes_gen_tables() like this:

FSb[0x00] = 0x63;
RSb[0x63] = 0x00;

for( i = 1; i < 256; i++ )
{
    x = pow[255 - log[i]];

    y = x;  y = ( y << 1 ) | ( y >> 7 );
    x ^= y; y = ( y << 1 ) | ( y >> 7 );
    x ^= y; y = ( y << 1 ) | ( y >> 7 );
    x ^= y; y = ( y << 1 ) | ( y >> 7 );
    x ^= y ^ 0x63;

    FSb[i] = x;
    RSb[x] = i;
}

The other initialization of Fsb[256] is easy to spot in the code file and is a static constant which is defined like this:

/* forward S-box */

static const uint32 FSb[256] =
{
    0x63, 0x7C, 0x77, 0x7B, 0xF2, 0x6B, 0x6F, 0xC5,
    0x30, 0x01, 0x67, 0x2B, 0xFE, 0xD7, 0xAB, 0x76,
    0xCA, 0x82, 0xC9, 0x7D, 0xFA, 0x59, 0x47, 0xF0,
    0xAD, 0xD4, 0xA2, 0xAF, 0x9C, 0xA4, 0x72, 0xC0,
    0xB7, 0xFD, 0x93, 0x26, 0x36, 0x3F, 0xF7, 0xCC,
    0x34, 0xA5, 0xE5, 0xF1, 0x71, 0xD8, 0x31, 0x15,
    0x04, 0xC7, 0x23, 0xC3, 0x18, 0x96, 0x05, 0x9A,

         < ... Truncated for readability ... >

    0x9B, 0x1E, 0x87, 0xE9, 0xCE, 0x55, 0x28, 0xDF,
    0x8C, 0xA1, 0x89, 0x0D, 0xBF, 0xE6, 0x42, 0x68,
    0x41, 0x99, 0x2D, 0x0F, 0xB0, 0x54, 0xBB, 0x16
};

---

Two initialization of Fsb[]?

Yes, the author of the program describes it in the comments:

/* uncomment the following line to use pre-computed tables */
/* otherwise the tables will be generated at the first run */

/* #define FIXED_TABLES */

#ifndef FIXED_TABLES

/* forward S-box & tables */
...
...
...

#else

/* forward S-box */

static const uint32 FSb[256] =
{
    0x63, 0x7C, 0x77, ...
...
...

Thus, if you want the array to be pre-determined, just uncomment #define FIXED_TABLES and you'll be fine.

---

Debugger time!

The functions which is responsible to generate the forward S-box is aes_gen_tables( void ) which is called from aes_set_key( ... ). The latter is called from the aescrypt2.c. We can spot the call to aes_set_key() in IDA:

From there, locating aes_gen_tables() is easy peasy:

As we said, the array is dynamically filled with values and we can view it using the debugger. Let's find the address of the array to be initialized and put a Hardware Breakpoint on Write on this address.

Here's the part where the first item of the Fsb is initialized:

This is the equivalent part from the source code:

/* generate the forward and reverse S-boxes */

FSb[0x00] = 0x63;
RSb[0x63] = 0x00;

Now put the Hardware Breakpoint on 0x00412F00:

Now we all set and we can start the debug session. Press the play button and start the program, then you'll have to input information to the console that has just popped up. After done, pressing Enter in the console will trigger our HW breakpoint. This is where Fsb[0] is set to 0x63. Press the play button and the breakpoint again will be triggered, this time it's at the end of the generation loop and our array will be filled:

There we go, now we found our forward S-box after it was dynamically generated by the program.