IDA stack pointer analysis fails due to "and" operation to esp

Question

I'm reversing an x86 executable with IDA pro. Here is the beginning of the function causing problems to IDA stack pointer analysis. I'm showing the stack pointer value in the second column:

.text:002C35B0 000                 push    ebp
.text:002C35B1 004                 mov     ebp, esp
.text:002C35B3 004                 and     esp, 0FFFFFFF8h
.text:002C35B6 004                 push    0FFFFFFFFh
.text:002C35B8 008                 push    offset sub_75B75D
.text:002C35BD 00C                 mov     eax, large fs:0

In the third line, the and is not changing the esp value for IDA and I suppose this is the reason for having unknown esp offsets throughout the function. I have the following questions:

• what is the purpose of this operation? Why not a normal sub or add here?
• what is the value of the stack pointer after that?

Answer 1

and esp, not 7 aligns the stack pointer (down) to a multiple of eight, by computing the moral equivalent of esp -= esp % 8; (which would require more instructions). You can manually change the simulated stack pointer by hitting Alt+K on the instruction and entering -4.

SP-related things in IDA tend to be a bit confusing because you never really know whether you are working in the realm of IDA's virtual inverted stack (as evidenced by the positive stack pointer offsets shown in the disassembly) or not... So you have to experiment, see what IDA does when you enter certain values.

Also, most things regarding simulation, constant/parameter propagation (this pointers!) and so on seem to have been reserved for the decompiler product, which might explain why IDA itself has seen little progress in that regard over the last decade.

Be that as it may, the small difference of -4 is probably not the reason you're seeing bad stack offsets. IDA sometimes gets confused when there are function chunks involved, or multiple code paths involving alloca() and so on (especially if the function epilogue is not located in the main chunk).

If you're still having problems you could put a bigger snippet on pastebin and post a link here. Also, if you have an active support plan then you could contact Hex-Rays; when I had a problem with stack pointers in a big IDB imported from an older version of IDA I sent in the IDB and they fixed it for me...