What does __return_ptr do in IDA?

Question

Let's say I have this program:

struct A {
    int x;
};

struct A func1(int z) {
    struct A result;
    result.x = z+1;
    return result;
}

int main() {
    return func1(1).x;
}

When compiling this on linux with gcc -m32 -fno-stack-protector the result of func1 will be returned as a stack pointer. Can I somehow set the type of func1 using __return_ptr so IDA understands this calling convention? I thought that would be the purpose of __return_ptr, if not what does it do?

Answer 1

The "problem" with your example is that the structure is too small (four bytes), so it fits in a register and is not actually passed on the stack. From the Itanium C++ ABI (used by most GCC implementations):

A type is considered non-trivial for the purposes of calls if:

• it has a non-trivial copy constructor, move constructor, or destructor, or
• all of its copy and move constructors are deleted.

This definition, as applied to class types, is intended to be the complement of the definition in [class.temporary]p3 of types for which an extra temporary is allowed when passing or returning a type. A type which is trivial for the purposes of the ABI will be passed and returned according to the rules of the underlying C ABI, e.g. in registers; often this has the effect of performing a trivial copy of the type.

(emphasis mine).

Since your struct is trivial and fits completely into the return register (eax), it is not passed on the stack but returned directly in eax. So there is no need for anything special here. However, if you make it a little bigger, e.g.:

struct A {
    int x;
    int y;
    int z;
};

Then it no longer fits and has to be passed on the stack. If you add the struct to Local Types and specify the C++ prototype for the function (struct A func1(int z)), then IDA performs type lowering and converts it to a C-style prototype with explicit argument locations:

struct A *__cdecl func1(struct A *__return_ptr __struct_ptr retstr, int z)

It seems currently __return_ptr is not used by IDA or decompiler and is just an annotation for the user, but in theory it could be used to make the decompiled code closer to the original C++ syntax.