4

tl;dr: I am reversing a GoLang 1.10 executable, compiled for Windows. I am trying to make IDA correctly recognize the calling convention.

Details: I am looking at an x86 executable where the following (nonstandard) calling convention is used.

  1. caller creates space on the stack for possibly multiple return values, i.e. subtracts 4*return_count from ESP.
  2. caller pushes arguments to callee to the stack, calls callee
  3. callee creates a stack frame for local variables
  4. callee does work
  5. callee restores ESP to be the same as before step 3
  6. callee stores return values on the stack below its arguments
  7. call returns

The caller now has the return values and the arguments still in the local stack frame, the latter on top of the former, and cleans up. This is a GoLang 1.10 executable, but I am describing the calling convention that I am seeing here, I do not know any reference for this. If something about the GoLang calling convention in Windows compiled executables is known; I would be very interested in that as well.

More to the point however, I am looking for a way to make IDA correctly understand this calling convention. As an example, consider the following code:

 test proc near                   ; test receives two arguments
   sub     esp, 18h               ; create stack frame
   mov     eax, [esp+1Ch]         ; argument 1 to test:
   mov     [esp], eax             ;  moved to top of stack frame
   mov     eax, [esp+20h]         ; argument 2 to test:
   mov     [esp+4], eax           ;  moved second to top of stack frame
   call    sub1                   ; call sub1(test_arg_1, test_arg_2)
   mov     eax, [esp+14h]         ; retrieve return_value_1
   mov     ecx, [esp+10h]         ; retrieve return_value_2
   mov     [esp], ecx           
   mov     [esp+4], eax
   call    sub2                   ; call sub2(return_value_1, return_value_2)
   movzx   eax, byte ptr [esp+8]  ; retrieve single return value
   xor     eax, 1                 ; negate result
   mov     [esp+24h], al          ; store result below arguments to test
   add     esp, 18h               ; close stack frame
   retn                           ; return

I tried to give sub1 more parameters:

int __cdecl sub1(int, int, int, int, int, int)

But unfortunately, it is not reflected in the decompiled code that the last two "arguments" to this function are actually return values which are being passed on to sub2:

int __cdecl test(int a1, int a2)
{
  int v2; // ST10_4
  int v3; // ST14_4
  unsigned __int8 v4; // ST08_1
  int esp_08; // [esp+8h] [ebp-10h]
  int esp_0C; // [esp+Ch] [ebp-Ch]
  int esp_10; // [esp+10h] [ebp-8h]
  int esp_14; // [esp+14h] [ebp-4h]
  void *retaddr; // [esp+18h] [ebp+0h]

  sub1(a1, a2, esp_08, esp_0C, esp_10, esp_14);
  sub2(v2, v3);
  return v4 ^ 1;
}

In an attempt to make use of this advanced calling convention syntax, I also created a structure containing two 32 bit integer fields and declared sub1 as follows:

struc_1 __usercall sub1@<0:^8.4,4:^12.4>(int, int)

I tried several variations of this, different offsets, also returning only a native type, but IDA 7.1 always crashed with the following message:

Oops! internal error 1004 occurred. Further work is not possible and IDA will close.

Is there any good way or workaround to make IDA recognize this calling convention and accurately reflect it in the decompiled code?

Reproduce MWE

To produce the above example, you can use the following test.go source file:

package main
import "os"

func sub2(x int, y int) bool { return x == y }
func sub1(x int, y int) (int, int) { return y,x }
func test(x int, y int) bool { return ! sub2(sub1(x,y)) }

func main() {
  test(1,2)
  os.Exit(0)
}

You can compile it as follows (disabling optimization is important):

set GOARCH=386
go build -gcflags=-N -gcflags=-l test.go

In IDA, the test function should be automatically renamed to main_test.

  • 2
    Since you appear to have the latest IDA and it crashes, you can probably contact support with this question. – mrexodia May 18 '18 at 11:18
  • A good idea, thanks. I was somewhat afraid to be using the syntax incorrectly, but I will ask them about it. – james May 18 '18 at 22:10
  • Does it work if you declare it like this? int __cdecl test(int a1, int a2, int * return1, int * return2) – Anton Kukoba May 22 '18 at 8:23
  • Dear @AntonKukoba, thanks for the idea, sadly it doesn't work. In that case, it just thinks that the arguments are int pointers rather than ints, nothing else changes. – james May 22 '18 at 9:34
  • Can you give me a sample, I'd like to experiment – Anton Kukoba May 22 '18 at 10:04
1

I have been in touch with IDA support and unfortunately, right now, there does not seem to be a good way to do this. IDA assumes in many cases that the return value of a function must be stored in a register. This is also the reason for the crash when trying to overrule this inherent assumption with scattered arguments.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.