Determining an application's library call behaviour

Question

I have a closed source Linux library libfoo that makes calls into another closed source library libbar for additional (mostly unneeded) features. However, I can't use libbar for ethical reasons, so I am trying to make a dummy library libfakebar that "implements" as little as possible of libbar while still being able to be used as a drop-in replacement (i.e., as close as possible to a collection of stubs).

I initially tried to simply extract the defined symbols from libbar via nm -D --defined-only libbar.so and only create stubs for those symbols, referencing the available public documentation for the appropriate function signatures. However, this fails when libfoo attempts to make calls that are not based on symbols. Specifically, it calls BarInstance() in libbar to get a pointer to a C++ class instance, adds a seemingly arbitrary value to that pointer, then uses the value found at the new address as a pointer to the next function it calls. Here is an example demonstrating this:

callq  0x7fff97158300 <BarInstance@plt>
mov    (%rax),%r8
lea    0x96c165(%rip),%rcx        # 0x7fff97b50e65
mov    %ebp,%edx
mov    %r12d,%esi
mov    %rax,%rdi
callq  *0x28(%r8)

I can't understand how this is supposed to make sense or be "legal" code. Remarkably, if I make BarInstance() return a collection of pointers to stub functions instead of a class instance, libfoo runs without complaints and everything "works". Unfortunately, it's not quite good enough to meet my minimum functionality requirements as there is at least one feature I need that doesn't work with said stubs, and the strange calling behaviour of libfoo doesn't lend itself towards helping me understand what more may need to be added where.

I'm not sure what impact this may make, but it appears as though libbar was built using GCC's -fvisibility=hidden. I have not been able to figure out exactly what to make of the above calling behaviour and as such what I need to do in libfakebar in order to properly replicate whatever is necessary for libfoo's code to work as expected - what's going on here?

Answer 1

This appears to be fairly typical virtual function call where the pointer to the virtual function table is stored in the object at offset 0000 and the call is to the function pointed to at offset 0028 in the table.

The object's virtual function table pointer (at offset 0000) is initialized in the class constructor. Since the constructor will be in the library, there does not need to be any direct reference to the virtual function table in the calling code. Similarly, a normal call to virtual function is via the virtual function table and there is no need for any direct reference to it in the calling code either.

For consistency between the application and library, it is important that both

• the library header file(s) used for the compilation of the application match the library binary, and
• the compiler used to compile the application uses the same Application Binary Interface as that used to compile the library.

For more information about this you might want to read about binary compatibility.

One of the things specified by an ABI is the order of the functions in the virtual function table. For example, the Itanium C++ ABI (used by GCC) says

"The order of the virtual function pointers in a virtual table is the order of declaration of the corresponding member functions in the class."

Hence, if the declaration order were to change between building the library and building the application there will no longer be binary compatibility.