I don't have experience of OSX, but at the least in Linux you can do this:
First, get some idea what system calls (
int 80h in Linux) the library might be using, if any. You can use your favorite disassembler or code tracer for this part. If the library itself does not have any system calls, find out or deduce what system calls its child functions may be using. If you have no idea or this step gets too difficult due to the size and/or complexity of the library, you can skip this first part.
gdb with the program. If by now you already know which system calls to catch, then catch only those. Otherwise catch them all. The command is
catch syscall. Then
r to run the program.
At this point I usually do
disp/10i $pc to see a little a bit of the code.
Once you're inside the system call, do
backtrace to print the stack frames. Set breakpoints in all of them or only in the ones you're interested of. The instruction to use is
break, for example
break *0xdeadbeef. Then
If you arrived in the desired library or function, great. If not, then
delete the breakpoint you arrived in and then
continue, and repeat until you arrive in the function you're interested of.
Hope this helps.