5

I have an executable that runs in the operating system, but when when I open it in $tool I get an error. What is going on?

Improve this question
7
  • 6
    An example of an error (e.g. a screenshot) would make this more realistic.
    – Igor Skochinsky ♦
    Commented Apr 18, 2013 at 18:29
  • Dunno why you were voted down. It's a worthy topic. Perhaps the question wasn't clear.
    – dyasta
    Commented Apr 19, 2013 at 8:31
  • 1
    @90h: the question is relatively clear, but Igor's point of criticism hasn't been addressed at all, but it would make sense to address it.
    – 0xC0000022L ♦
    Commented Apr 19, 2013 at 14:38
  • 1
    @VehmStark Let the community decide. I think it's fine as is. It is a general question with a general answer, and doesn't require specifics to have utility.
    – dyasta
    Commented Apr 21, 2013 at 11:22
  • 1
    A related question at RE.SE, including a couple specific examples, is here: How to Prevent Use of Resource Editors
    – dyasta
    Commented Apr 21, 2013 at 11:43

2 Answers 2

Reset to default
9

This is caused by a differences in the operating system's loader and the file format parsing code in the tool you are using. Malicious program authors often exploit differences between an executable file format's specification and how the file format is actually used by the loader in practice. If there are differences between the file format specification and the operating system loader, there exist executables which will run but are not legal according to the specification! For example, up until Windows Vista a PE executable can be missing several header fields and the Window's loader will still load it. It's likely your tool was written by looking at the file format specification and not at the implementation of the loader and thus cannot necessarily handle all executables which are successfully loaded by the operating system.

In reality an executable file format is specified in three categories of places:

  1. the formal file format specification
  2. the implementation of the operating system's loader
  3. the implementation of third party tools

These three things often differ in very subtle ways and malicious program authors take advantage of this fact. Just remember that the ultimate arbiter for whether something is a valid executable or not is the loader. If the executable runs but your tools are failing, your tools do not take into account the real-world behavior of the operating system loader.

The best work documenting this phenomena that I am aware of is by Ange Albertini for the PE executable file format.

Improve this answer
1
  • 2
    After Windows Vista, it's not possible anymore to cut the OptionalHeader at Subsystem: the comment in pefile is only valid until Windows XP.
    – Ange
    Commented Apr 18, 2013 at 20:20
1

Try using Rohitab's API Monitor to use the loader to open the program and watch the failed calls. Might point you in the right direction.

Improve this answer
1
  • 1
    actually this wouldn't help, because the third-party programs handling PE files will not use the default library functions provided by the operating system, e.g. as part of ntdll.dll.
    – 0xC0000022L ♦
    Commented Apr 22, 2013 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.