4

I have an executable that runs in the operating system, but when when I open it in $tool I get an error. What is going on?

  • 6
    An example of an error (e.g. a screenshot) would make this more realistic. – Igor Skochinsky Apr 18 '13 at 18:29
  • Dunno why you were voted down. It's a worthy topic. Perhaps the question wasn't clear. – dyasta Apr 19 '13 at 8:31
  • 1
    @90h: the question is relatively clear, but Igor's point of criticism hasn't been addressed at all, but it would make sense to address it. – 0xC0000022L Apr 19 '13 at 14:38
  • 1
    @VehmStark Let the community decide. I think it's fine as is. It is a general question with a general answer, and doesn't require specifics to have utility. – dyasta Apr 21 '13 at 11:22
  • 1
    A related question at RE.SE, including a couple specific examples, is here: How to Prevent Use of Resource Editors – dyasta Apr 21 '13 at 11:43
9

This is caused by a differences in the operating system's loader and the file format parsing code in the tool you are using. Malicious program authors often exploit differences between an executable file format's specification and how the file format is actually used by the loader in practice. If there are differences between the file format specification and the operating system loader, there exist executables which will run but are not legal according to the specification! For example, up until Windows Vista a PE executable can be missing several header fields and the Window's loader will still load it. It's likely your tool was written by looking at the file format specification and not at the implementation of the loader and thus cannot necessarily handle all executables which are successfully loaded by the operating system.

In reality an executable file format is specified in three categories of places:

  1. the formal file format specification
  2. the implementation of the operating system's loader
  3. the implementation of third party tools

These three things often differ in very subtle ways and malicious program authors take advantage of this fact. Just remember that the ultimate arbiter for whether something is a valid executable or not is the loader. If the executable runs but your tools are failing, your tools do not take into account the real-world behavior of the operating system loader.

The best work documenting this phenomena that I am aware of is by Ange Albertini for the PE executable file format.

  • 2
    After Windows Vista, it's not possible anymore to cut the OptionalHeader at Subsystem: the comment in pefile is only valid until Windows XP. – Ange Apr 18 '13 at 20:20
1

Try using Rohitab's API Monitor to use the loader to open the program and watch the failed calls. Might point you in the right direction.

  • 1
    actually this wouldn't help, because the third-party programs handling PE files will not use the default library functions provided by the operating system, e.g. as part of ntdll.dll. – 0xC0000022L Apr 22 '13 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.