Stack-pointer analysis failed

Question

I'm testing my toy obfuscating C compiler against IDA's decompiler, but IDA refuses always functions (then the decompilation is not possible). For example, at one of lowest-levels of obfuscating, a function likes

int foo(int i) {
  return i;
}

is compiled to

0x0         40 55                          push rbp
0x2         48 89 e5                       mov rbp, rsp
0x5         41 56                          push r14
0x7         41 57                          push r15
0x9         48 83 ec 10                    sub rsp, 0x10
0xd         40 89 bc 24 0c 00 00 00        mov [rsp+0xc], edi
0x15        48 8d 05 e4 ff ff ff           lea rax, [rip-0x1c]
0x1c        44 8b bc 24 0c 00 00 00        mov r15d, [rsp+0xc]
0x24        44 8b b4 24 0c 00 00 00        mov r14d, [rsp+0xc]
0x2c        45 0f af fe                    imul r15d, r14d
0x30        44 89 f9                       mov ecx, r15d
0x33        83 c1 01                       add ecx, 0x1
0x36        44 8b b4 24 0c 00 00 00        mov r14d, [rsp+0xc]
0x3e        45 0f af f7                    imul r14d, r15d
0x42        48 63 c9                       movsxd rcx, ecx
0x45        49 63 d6                       movsxd rdx, r14d
0x48        48 39 d1                       cmp rcx, rdx
0x4b        0f 95 c1                       setnz cl
0x4e        40 84 c9                       test cl, cl
0x51        75 1c                          jnz 0x1c
0x53        44 8b bc 24 0c 00 00 00        mov r15d, [rsp+0xc]
0x5b        49 63 cf                       movsxd rcx, r15d
0x5e        44 8b bc 24 0c 00 00 00        mov r15d, [rsp+0xc]
0x66        44 89 ff                       mov edi, r15d
0x69        48 89 c6                       mov rsi, rax
0x6c        40 ff d1                       call rcx
0x6f        44 8b bc 24 0c 00 00 00        mov r15d, [rsp+0xc]
0x77        44 89 f8                       mov eax, r15d
0x7a        48 83 c4 10                    add rsp, 0x10
0x7e        41 5f                          pop r15
0x80        41 5e                          pop r14
0x82        40 5d                          pop rbp
0x84        c3                             ret

I still don't understand why the stack frame analysis can be failed for this case. There are only two groups of instructions modifying rsp, the first 5:

0x0         40 55                          push rbp
0x2         48 89 e5                       mov rbp, rsp
0x5         41 56                          push r14
0x7         41 57                          push r15
0x9         48 83 ec 10                    sub rsp, 0x10

and the last 5:

0x7a        48 83 c4 10                    add rsp, 0x10
0x7e        41 5f                          pop r15
0x80        41 5e                          pop r14
0x82        40 5d                          pop rbp
0x84        c3                             ret

and they actually make the stack pointer balanced.

How can I fix that?

Update: @chentiangemalc suggests that it comes from call rcx, then I changed the obfuscating option to not generate such a call, and removed almost all other obfuscating options, following is an even simpler result

0x0         40 55                          push rbp
0x2         48 89 e5                       mov rbp, rsp
0x5         40 53                          push rbx
0x7         48 83 ec 08                    sub rsp, 0x8
0xb         c1 ef 00                       shr edi, 0x0
0xe         89 f8                          mov eax, edi
0x10        81 c8 33 19 d0 39              or eax, 0x39d01933
0x16        89 f9                          mov ecx, edi
0x18        81 e1 33 19 d0 39              and ecx, 0x39d01933
0x1e        0f af c1                       imul eax, ecx
0x21        40 b9 33 19 d0 39              mov ecx, 0x39d01933
0x27        89 ca                          mov edx, ecx
0x29        f7 d2                          not edx
0x2b        89 fb                          mov ebx, edi
0x2d        21 d3                          and ebx, edx
0x2f        f7 d7                          not edi
0x31        21 f9                          and ecx, edi
0x33        0f af d9                       imul ebx, ecx
0x36        01 d8                          add eax, ebx
0x38        81 c0 31 d6 93 7f              add eax, 0x7f93d631
0x3e        40 b9 fb 89 99 8c              mov ecx, 0x8c9989fb
0x44        0f af c1                       imul eax, ecx
0x47        81 c0 f5 c4 23 fd              add eax, -0x2dc3b0b
0x4d        48 8d 8c 24 04 00 00 00        lea rcx, [rsp+0x4]
0x55        40 89 01                       mov [rcx], eax
0x58        48 8d 84 24 04 00 00 00        lea rax, [rsp+0x4]
0x60        40 8b 00                       mov eax, [rax]
0x63        48 83 c4 08                    add rsp, 0x8
0x67        40 5b                          pop rbx
0x69        40 5d                          pop rbp
0x6b        c3                             ret

but IDA complains always sp-analysis failed ((I've put the ELF here).

Other tools like Ghidra or JEB Decompiler happily recognizes the function, though.

Update: the method of @chentiangemalc works perfectly. for the example above, unfortunately there are cases where IDA is not happy (sample: ELF).

Answer 1

When I assembled the code, the offending instruction was:

0x6c        40 ff d1                       call rcx

You will need to use Change Stack Pointer command in IDA to fix this in disassembly. As per IDA documentation:

• This command allows you to specify how the stack pointer (SP) is modified by the current instruction.
• You cannot use this command if the current instruction does not belong to any function.
• You will need to use this command only if IDA was not able to trace the value of the SP register.
• Usually IDA can handle it but in some special cases it fails. An example of such a situation is an indirect call of a function that purges its parameters from the stack. In this case, IDA has no information about the function and cannot properly trace the value of SP.
• Please note that you need to specify the difference between the old and new values of SP.
• The value of SP is used if the current function accesses local variables by [ESP+xxx] notation.

This can also be verified by removing the call rcx instruction and confirming it removes the SP Analysis failed error in IDA.

Based on the 2nd version with uploaded ELF the problem is IDA hasn't detected the end of the function correctly.

Right clicking the function foo in Functions view and selecting Edit function and changing the end address to 0:000000000000006C fixes the SP analysis problem.

When trouble shooting SP Analysis failures also enabling Stack Pointer in Options | General and selecting Stack Pointer can help you determine cause of problems, this will show the stack pointer value in green text to the left of instructions.

The 3rd sample provided it's the same issue. To find the end of function, switch to Text view find the offset of retn and add one, for example the end of function in the last sample provided is 5D5.

The root cause of the problem is the redundant REX prefix (0x40) at the start. Normal 64-bit function prologs start with 55 48 89 E5 so IDA tries to create a function at address 1 which interferes with the actual function at 0 and makes it stop after one instruction. This planned to be fixed in future releases of IDA.