What I have so far:
- Application that injects my shared object into another process via ptrace
- shared object hook that can hook functions like strcmp
What I'm trying to do:
- call non-exported internal functions of the shared library
- hook these functions
So far I've tried using tools like nm/objdump to dump the assembly. I've grabbed the mangled function names, and I'm attempting to call some of these functions. I've attempted to locate the function by using dlsym inside the hook, and it gives me an address. I've tried assigning the function pointer directly to that address, as well as adding that address to the shared object offset, but neither is returning properly.
I guess my first question is am I even going down the right path here trying to use dlsym? If I don't know the correct return type, and call the function what behavior would i be expecting to see? I was expecting an app crash but I'm not seeing that, which is making me wonder if the function is actually being called.
There's no java involved in this if it matters. Everything is c and the shared library is c++ app using a common framework.