Figure out whether a function has return value of not?

Question

I am doing binary analysis on x86-64bit ELF binaries. All the binaries are compiled from C language. Basically, for a given function, I would like to figure out whether this function has a return value or not. That is, in its corresponding C code, whether a meaningful return exists.

As I am essentially facing the assembly code, it is not feasible to figure out through some type information. However, as for normal x86-64bit assembly program, the calling convention only allows register rax to hold the return value, so I am thinking to check the usage of rax after a typical function call and decide whether the target function returns a value.

Here is an example in AT&T syntax:

foo:
   ...
   call bar
   mov 0, %rax  <--- bar should not have a return value

bar:
   ...

In the above example, as rax is immediately reset, it is unlikely for function bar to return a value.

Another example:

foo:
   ...
   call bar
   jmp *%rax  <----- It is very likely that bar has a return value

For the above case, I suppose without some aggressive inter-procedure optimization, we can say it for sure that bar returns a value (a pointer).

I think this is yet another (ad-hoc) reverse engineering task, but I guess there may be a more "formal" way to solve it, any idea on that?

Answer 1

I see several problems with that approach:

• Functions that return double values don't use eax
• Functions that return structs don't neccesarily use eax, see here
• return values from many functions, like free, close, printf are custumarily ignored, so "caller does not read eax" does not translate to "function has no return value"
• there are edge cases to "eax is used". For example, xor [location], eax probably means eax has a value, xor eax, [location] as well, but xor eax, eax means probably not.

In the generic case, i think there's no foolproof way. For example, in a function ending in

for (i=0; i<somevar; i++)
    somearray[i]=0;
return i;

the compiler may just decide to use eax for the loop counter; which means there's no reason to do another mov after the loop. If the caller ignores the value of eax, you have no way to determine, from the assembly alone, if that return statement was present or not. (Of course, in this particular case, any self respecting compiler will generate a variant of rep stosw or sse instructions, but you get the point).

So when the caller does read eax, you can be quite certain that the function has a return value; but a caller that ignores eax basically means nothing.

And even if the caller reads eax, you could construct pathological cases of a function written in assembler, that preserves eax, and a caller that knows about this and makes use of eax even over the function call. But you'll probably not encounter this in software that isn't deliberately obfuscated

Answer 2

The return value is defined by the calling convention used by your compiler, and it usually just means "the value of eax/rax".

In your first example, rax is not reset, it's just set to the value 0, hence the functions most likely ends with a return 0;.

In your second example, you should check what is the last write on rax in the bar function.

You could try to check whether the next operation on rax after your function is a write or a read, basically if it's a read the return value is used, if it's a write the return value is discarded.