How to specify stack-based return values in IDA Pro (GoLang)

Question

tl;dr: I am reversing a GoLang 1.10 executable, compiled for Windows. I am trying to make IDA correctly recognize the calling convention.

Details: I am looking at an x86 executable where the following (nonstandard) calling convention is used.

1. caller creates space on the stack for possibly multiple return values, i.e. subtracts 4*return_count from ESP.
2. caller pushes arguments to callee to the stack, calls callee
3. callee creates a stack frame for local variables
4. callee does work
5. callee restores ESP to be the same as before step 3
6. callee stores return values on the stack below its arguments
7. call returns

The caller now has the return values and the arguments still in the local stack frame, the latter on top of the former, and cleans up. This is a GoLang 1.10 executable, but I am describing the calling convention that I am seeing here, I do not know any reference for this. If something about the GoLang calling convention in Windows compiled executables is known; I would be very interested in that as well.

More to the point however, I am looking for a way to make IDA correctly understand this calling convention. As an example, consider the following code:

 test proc near                   ; test receives two arguments
   sub     esp, 18h               ; create stack frame
   mov     eax, [esp+1Ch]         ; argument 1 to test:
   mov     [esp], eax             ;  moved to top of stack frame
   mov     eax, [esp+20h]         ; argument 2 to test:
   mov     [esp+4], eax           ;  moved second to top of stack frame
   call    sub1                   ; call sub1(test_arg_1, test_arg_2)
   mov     eax, [esp+14h]         ; retrieve return_value_1
   mov     ecx, [esp+10h]         ; retrieve return_value_2
   mov     [esp], ecx           
   mov     [esp+4], eax
   call    sub2                   ; call sub2(return_value_1, return_value_2)
   movzx   eax, byte ptr [esp+8]  ; retrieve single return value
   xor     eax, 1                 ; negate result
   mov     [esp+24h], al          ; store result below arguments to test
   add     esp, 18h               ; close stack frame
   retn                           ; return

I tried to give sub1 more parameters:

int __cdecl sub1(int, int, int, int, int, int)

But unfortunately, it is not reflected in the decompiled code that the last two "arguments" to this function are actually return values which are being passed on to sub2:

int __cdecl test(int a1, int a2)
{
  int v2; // ST10_4
  int v3; // ST14_4
  unsigned __int8 v4; // ST08_1
  int esp_08; // [esp+8h] [ebp-10h]
  int esp_0C; // [esp+Ch] [ebp-Ch]
  int esp_10; // [esp+10h] [ebp-8h]
  int esp_14; // [esp+14h] [ebp-4h]
  void *retaddr; // [esp+18h] [ebp+0h]

  sub1(a1, a2, esp_08, esp_0C, esp_10, esp_14);
  sub2(v2, v3);
  return v4 ^ 1;
}

In an attempt to make use of this advanced calling convention syntax, I also created a structure containing two 32 bit integer fields and declared sub1 as follows:

struc_1 __usercall sub1@<0:^8.4,4:^12.4>(int, int)

I tried several variations of this, different offsets, also returning only a native type, but IDA 7.1 always crashed with the following message:

Oops! internal error 1004 occurred. Further work is not possible and IDA will close.

Is there any good way or workaround to make IDA recognize this calling convention and accurately reflect it in the decompiled code?

Reproduce MWE

To produce the above example, you can use the following test.go source file:

package main
import "os"

func sub2(x int, y int) bool { return x == y }
func sub1(x int, y int) (int, int) { return y,x }
func test(x int, y int) bool { return ! sub2(sub1(x,y)) }

func main() {
  test(1,2)
  os.Exit(0)
}

You can compile it as follows (disabling optimization is important):

set GOARCH=386
go build -gcflags=-N -gcflags=-l test.go

In IDA, the test function should be automatically renamed to main_test.

Answer 1

I have been in touch with IDA support and unfortunately, right now, there does not seem to be a good way to do this. IDA assumes in many cases that the return value of a function must be stored in a register. This is also the reason for the crash when trying to overrule this inherent assumption with scattered arguments.