I am doing binary analysis on x86-64bit
ELF binaries. All the binaries are compiled from C
language. Basically, for a given function, I would like to figure out whether this function has a return value or not. That is, in its corresponding C
code, whether a meaningful return
exists.
As I am essentially facing the assembly code, it is not feasible to figure out through some type information. However, as for normal x86-64bit
assembly program, the calling convention only allows register rax
to hold the return value, so I am thinking to check the usage of rax
after a typical function call and decide whether the target function returns a value.
Here is an example in AT&T
syntax:
foo:
...
call bar
mov 0, %rax <--- bar should not have a return value
bar:
...
In the above example, as rax
is immediately reset, it is unlikely for function bar
to return a value.
Another example:
foo:
...
call bar
jmp *%rax <----- It is very likely that bar has a return value
For the above case, I suppose without some aggressive inter-procedure optimization, we can say it for sure that bar
returns a value (a pointer).
I think this is yet another (ad-hoc) reverse engineering task, but I guess there may be a more "formal" way to solve it, any idea on that?