Why doesn't this retn instruction return from the function?

Question

I disassembled an exe file on Windows with IDA and saw this function prologue:

I see that when the retn executes instead of returning it jumps to sub_1314cba4.

1. Why doesn't the retn instruction actually return? I thought returning in the entry point will always exit the process.

2. Why isn't the stack damaged? The function doesn't allocate any stack space using sub esp, 0x... for example.

3. Is this behavior strange?

Answer 1

1. The address of sub_1313cba4 is moved to PC/IP but not directly, as this cannot be done. It is done by pushing the address to the stack and relying on the fact that retn will get what's on the top of the stack and redirect the execution there.

2. Well, it depends on what you mean by damaging the stack. The stack is just a place in memory and the damage can only occur by not correctly handling data on it. For example, trying to redirect execution to an address that doesn't represent the code. If an application is built with keeping in mind what's on the stack at any moment it can correctly continue execution.

3. It is considered strange in normal applications, as this is not the normal way. If we are talking about crackmes, malware or any obfuscated binaries, then this is standard practice. It is done to make the reversing process more complex as calling functions this way is harder for tools to follow. It also makes harder for the reverser to follow the application's logic.

Answer 2

My answers:

1. Why is that? The program you are looking at is probably using some software obfuscation because this is not a usual way to return from a function. My guess is that this function is just here to break automatic binary analysis.

   Now, what does happen precisely here... In fact, this is quite simple, the address of sub_1313cba4 is pushed on to the stack and the ret instruction is executed. Basically, ret will just do a pop eip (beware, such instruction does not exist as we cannot address the eip register in i386). So, it will load the address of the subroutine sub_1313cba4 into eip and will start executing it (the memory context, I mean the stack, will remain as it is now. No new stack-frame will be created, so the subroutine must take this into account to work properly).

   This way of doing will totally lose the track of the call-stack by not following the rules (and will disturb many binary analysis tools such as IDA).

2. How comes this function do not damage the stack? This function does damage the [call] stack. It will be very hard to recover the call-stack starting from this point. But, it might be intended by the programmer of the software. As I was telling before, I really suspect some software obfuscation here.

3. Is that behavior strange? Yes, this is totally non-standard and probably intended to be.