1

I am doing binary analysis on x86-64bit ELF binaries. All the binaries are compiled from C language. Basically, for a given function, I would like to figure out whether this function has a return value or not. That is, in its corresponding C code, whether a meaningful return exists.

As I am essentially facing the assembly code, it is not feasible to figure out through some type information. However, as for normal x86-64bit assembly program, the calling convention only allows register rax to hold the return value, so I am thinking to check the usage of rax after a typical function call and decide whether the target function returns a value.

Here is an example in AT&T syntax:

foo:
   ...
   call bar
   mov 0, %rax  <--- bar should not have a return value

bar:
   ...

In the above example, as rax is immediately reset, it is unlikely for function bar to return a value.

Another example:

foo:
   ...
   call bar
   jmp *%rax  <----- It is very likely that bar has a return value

For the above case, I suppose without some aggressive inter-procedure optimization, we can say it for sure that bar returns a value (a pointer).

I think this is yet another (ad-hoc) reverse engineering task, but I guess there may be a more "formal" way to solve it, any idea on that?

2 Answers 2

3

I see several problems with that approach:

  • Functions that return double values don't use eax
  • Functions that return structs don't neccesarily use eax, see here
  • return values from many functions, like free, close, printf are custumarily ignored, so "caller does not read eax" does not translate to "function has no return value"
  • there are edge cases to "eax is used". For example, xor [location], eax probably means eax has a value, xor eax, [location] as well, but xor eax, eax means probably not.

In the generic case, i think there's no foolproof way. For example, in a function ending in

for (i=0; i<somevar; i++)
    somearray[i]=0;
return i;

the compiler may just decide to use eax for the loop counter; which means there's no reason to do another mov after the loop. If the caller ignores the value of eax, you have no way to determine, from the assembly alone, if that return statement was present or not. (Of course, in this particular case, any self respecting compiler will generate a variant of rep stosw or sse instructions, but you get the point).

So when the caller does read eax, you can be quite certain that the function has a return value; but a caller that ignores eax basically means nothing.

And even if the caller reads eax, you could construct pathological cases of a function written in assembler, that preserves eax, and a caller that knows about this and makes use of eax even over the function call. But you'll probably not encounter this in software that isn't deliberately obfuscated

2
  • Thank you Guntram. This is a fantastic answer. I learned a lot from it! I guess I will start from checking the usage of eax. Although when considering potential control flow transfer following the call instruction, tracing the usage of eax is by no means as easy as it looks... Apr 21, 2016 at 19:45
  • As I want to implement it like a dataflow analysis task. Apr 21, 2016 at 19:46
0

The return value is defined by the calling convention used by your compiler, and it usually just means "the value of eax/rax".

In your first example, rax is not reset, it's just set to the value 0, hence the functions most likely ends with a return 0;.

In your second example, you should check what is the last write on rax in the bar function.

You could try to check whether the next operation on rax after your function is a write or a read, basically if it's a read the return value is used, if it's a write the return value is discarded.

2
  • "You could try to check whether the next operation on rax after your function is a write or a read" - that's what he's doing. The first example has a write to rax after the call to bar, the second example reads rax. Apr 21, 2016 at 18:09
  • @GuntramBlohm. Hi, thank you for your reply. Yes, that's basically what I am thinking to do right now ;) Apr 21, 2016 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.