How to programmatically identify binary calling convention?

Question

I have a problem where I'm instrumenting stripped binaries; I don't know the start of main(). But there's always an init(), and init() calls libc_start_main(), which receives a pointer to main.

If I can instrument libc with analysis code to intercept the argument, then I can retrieve that address and place another pin callback there so that I can get it's arguments. The problem is, I don't know what the calling convention is; I was thinking, if I could boil this down to a matter of the calling convention, then I do this for any function. I did notice that gdb knows the calling convention of libc_start_main(), in fact it is so good, it knows the order of the arguments as well.

I did read a short note on stackoverflow that stated that the name of the function would yield the calling convention: « How to find the calling convention of a third party dll? »

If it's not possible to know the calling convention programmatically, what is the opinion on creating a local build of libc in order to be able to force a particular calling convention onto __libc_start_main()... you see my chain of thought. Does anybody think that this is a better approach, rather than solving it in the general case ?

Answer 1

Since you mention __libc_start_main it seems you need to worry only about Linux. In such case, the calling convention is known and fixed. In most cases it will be SystemV ABI or similar.

Answer 2

I found that (at least on Linux) it was reliable to intercept _init and to read from a constant offset from the stack pointer was pretty reliable. I ended up producing a pintool that would do just that.