Assuming that library X is statically linked, is there a way to know with an acceptable grade of confidence if a function from it was used in the binary?
I am currently using this approach:
- Convert the library function from its disassembly to its abstract representation to retrieve control flow structures, function and system calls and save them into a data structure by the order they appear in.
- Repeat the process with every statically linked function in the binary.
- Try to find similarities between the representation of the library function I am interested in and those found in the binary.
It works, but it can probably be done better and I fear I am missing code that uses particular compiler settings and/or obfuscation. For example, if an optimization flag is set and two or more functions are joined in the same monolithic piece of code, my approach will miss it. Also, it's not much time-efficient, which becomes a problem when I need to analyze a large binary against hundreds or thousands of libraries.
I was thinking of generating the AST for the entire binary instead of individual functions, and match parts of it with the library function's representation. Is there a general technique to do this, possibly already implemented in a publicly available tool? Is there a better idea?
I already know of the tools to generate abstract representations. I am more interested in the matching part, or ideally, in a tool that does both.
EDIT: @HamZa from security.SE has mentioned me IDA FLIRT and Zignatures. Are these effective against obfuscation/optimization?
EDIT 2: @HamZa has mentioned the difficulty to build an AST in some cases and proposed a hybrid approach, like symbolic execution that unfortunately, doesn't scale well. Also, he pointed me to this and this paper.
EDIT 3: this may be an option, but it's not for closed-source binaries (even though assembly code for certain architectures is covered). Also, is being offered as an Internet service and I'd rather not take it down by doing thousands of checks for different libraries every second.