I have a closed source Linux library libfoo that makes calls into another closed source library libbar for additional (mostly unneeded) features. However, I can't use libbar for ethical reasons, so I am trying to make a dummy library libfakebar that "implements" as little as possible of libbar while still being able to be used as a drop-in replacement (i.e., as close as possible to a collection of stubs).
I initially tried to simply extract the defined symbols from libbar via
nm -D --defined-only libbar.so and only create stubs for those symbols, referencing the available public documentation for the appropriate function signatures. However, this fails when libfoo attempts to make calls that are not based on symbols. Specifically, it calls
BarInstance() in libbar to get a pointer to a C++ class instance, adds a seemingly arbitrary value to that pointer, then uses the value found at the new address as a pointer to the next function it calls. Here is an example demonstrating this:
callq 0x7fff97158300 <BarInstance@plt>
lea 0x96c165(%rip),%rcx # 0x7fff97b50e65
I can't understand how this is supposed to make sense or be "legal" code. Remarkably, if I make
BarInstance() return a collection of pointers to stub functions instead of a class instance, libfoo runs without complaints and everything "works". Unfortunately, it's not quite good enough to meet my minimum functionality requirements as there is at least one feature I need that doesn't work with said stubs, and the strange calling behaviour of libfoo doesn't lend itself towards helping me understand what more may need to be added where.
I'm not sure what impact this may make, but it appears as though libbar was built using GCC's
-fvisibility=hidden. I have not been able to figure out exactly what to make of the above calling behaviour and as such what I need to do in libfakebar in order to properly replicate whatever is necessary for libfoo's code to work as expected - what's going on here?