How to find if a given library function is used in a binary?

Question

Assuming that library X is statically linked, is there a way to know with an acceptable grade of confidence if a function from it was used in the binary?

I am currently using this approach:

1. Convert the library function from its disassembly to its abstract representation to retrieve control flow structures, function and system calls and save them into a data structure by the order they appear in.
2. Repeat the process with every statically linked function in the binary.
3. Try to find similarities between the representation of the library function I am interested in and those found in the binary.

It works, but it can probably be done better and I fear I am missing code that uses particular compiler settings and/or obfuscation. For example, if an optimization flag is set and two or more functions are joined in the same monolithic piece of code, my approach will miss it. Also, it's not much time-efficient, which becomes a problem when I need to analyze a large binary against hundreds or thousands of libraries.

I was thinking of generating the AST for the entire binary instead of individual functions, and match parts of it with the library function's representation. Is there a general technique to do this, possibly already implemented in a publicly available tool? Is there a better idea?

I already know of the tools to generate abstract representations. I am more interested in the matching part, or ideally, in a tool that does both.

EDIT: @HamZa from security.SE has mentioned me IDA FLIRT and Zignatures. Are these effective against obfuscation/optimization?

EDIT 2: @HamZa has mentioned the difficulty to build an AST in some cases and proposed a hybrid approach, like symbolic execution that unfortunately, doesn't scale well. Also, he pointed me to this and this paper.

EDIT 3: this may be an option, but it's not for closed-source binaries (even though assembly code for certain architectures is covered). Also, is being offered as an Internet service and I'd rather not take it down by doing thousands of checks for different libraries every second.

Answer 1

As long as you can find a static library for a similar version of libc, you can use IDA Pro's FLIRT signatures.

Generally this code is not obfuscated, as the system's libc.a will be used. It's possible to compile libc from source and use obfuscations, but this is not what you'll usually encounter.

I've got a script (link below) that I use to generate a ".sig" file which you can drop in IDA Pro's sig/pc (or sig/arm or whatever is most relevant) and load it with FLIRT Signatures via File > Load File > FLIRT Signatures.

On macOS, the directory is e.g.: /Applications/IDA Pro 7.5/ida.app/Contents/MacOS/sig/pc

Once the FLIRT signatures are loaded, you can look for whatever symbols you're interested in from IDA.

[1]: https://gist.github.com/zachriggle/130e01cf4541615e2062ca66d1637a07