I have an executable that runs in the operating system, but when when I open it in
$tool I get an error. What is going on?
This is caused by a differences in the operating system's loader and the file format parsing code in the tool you are using. Malicious program authors often exploit differences between an executable file format's specification and how the file format is actually used by the loader in practice. If there are differences between the file format specification and the operating system loader, there exist executables which will run but are not legal according to the specification! For example, up until Windows Vista a PE executable can be missing several header fields and the Window's loader will still load it. It's likely your tool was written by looking at the file format specification and not at the implementation of the loader and thus cannot necessarily handle all executables which are successfully loaded by the operating system.
In reality an executable file format is specified in three categories of places:
- the formal file format specification
- the implementation of the operating system's loader
- the implementation of third party tools
These three things often differ in very subtle ways and malicious program authors take advantage of this fact. Just remember that the ultimate arbiter for whether something is a valid executable or not is the loader. If the executable runs but your tools are failing, your tools do not take into account the real-world behavior of the operating system loader.
Try using Rohitab's API Monitor to use the loader to open the program and watch the failed calls. Might point you in the right direction.