2

Let's say I have this program:

struct A {
    int x;
};

struct A func1(int z) {
    struct A result;
    result.x = z+1;
    return result;
}

int main() {
    return func1(1).x;
}

When compiling this on linux with gcc -m32 -fno-stack-protector the result of func1 will be returned as a stack pointer. Can I somehow set the type of func1 using __return_ptr so IDA understands this calling convention? I thought that would be the purpose of __return_ptr, if not what does it do?

  • question body seems to be different from title – Igor Skochinsky Jun 3 '18 at 14:10
  • rephrased question body – Stefan Friesel Jun 3 '18 at 16:02
  • you're still asking two different things: "what is __return_ptr?", and "how to represent this calling convention in IDA?". But okay, I'll try to describe both parts. – Igor Skochinsky Jun 3 '18 at 17:26
2

The "problem" with your example is that the structure is too small (four bytes), so it fits in a register and is not actually passed on the stack. From the Itanium C++ ABI (used by most GCC implementations):

A type is considered non-trivial for the purposes of calls if:

  • it has a non-trivial copy constructor, move constructor, or destructor, or
  • all of its copy and move constructors are deleted.

This definition, as applied to class types, is intended to be the complement of the definition in [class.temporary]p3 of types for which an extra temporary is allowed when passing or returning a type. A type which is trivial for the purposes of the ABI will be passed and returned according to the rules of the underlying C ABI, e.g. in registers; often this has the effect of performing a trivial copy of the type.

(emphasis mine).

Since your struct is trivial and fits completely into the return register (eax), it is not passed on the stack but returned directly in eax. So there is no need for anything special here. However, if you make it a little bigger, e.g.:

struct A {
    int x;
    int y;
    int z;
};

Then it no longer fits and has to be passed on the stack. If you add the struct to Local Types and specify the C++ prototype for the function (struct A func1(int z)), then IDA performs type lowering and converts it to a C-style prototype with explicit argument locations:

struct A *__cdecl func1(struct A *__return_ptr __struct_ptr retstr, int z)

It seems currently __return_ptr is not used by IDA or decompiler and is just an annotation for the user, but in theory it could be used to make the decompiled code closer to the original C++ syntax.

  • At least on my machine gcc does pass the result on the stack even with the 4-byte struct (gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0). But the last sentence answers my question – Stefan Friesel Jun 4 '18 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.