Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It only takes a minute to sign up.
Sign up to join this community
Now I'm developing an unpacker and reached testing stage. Sometimes I got a message which says that unpacked file is invalid Win32 executable, however it is not informative.
I would like to ask if there is a tool which identifies reason, why Windows exe's loader "doesn't like" file. So I would be able to easily detect bug in unpacking scripts.
There are many PE/header dumpers out there but as far as I know none is suitable for vetting an image; their goal is usually the extraction and display of information, not validation.
The best approach would be to write a small program or script in your favourite Swiss Army Knife language for validating the integrity of your output image, with a focus on flexibility and clarity in the section where you define your rules. In many languages there are libraries that you can call on, like pefile in python.
There is no place on the 'net where all the rules concerning PE images are collected in a comprehensive fashion. Many rules are not enforced by current incarnations of the Windows loader, or at least not consistently, even though they are mentioned somewhere on MSDN or in the PE COFF specification. Consequently, many packers and protectors violates certain existing rules with impunity. For the moment.
By the same token, if your output image happens to run on your current incarnation of Windows there's no guarantee that it will do so on another copy of Windows with a different version, patch level or bitness. Let alone on WINE or ReactOS, whatever.
Excellent sources of information - besides Google - are the ReactOS project and the OS dev wiki.
Two things to look out for specifically - besides correctness of the header info, the data directory and the section table - are the image checksum and file/section alignment. Also, for some tasks - like computing the checksum - you can call upon functions in Microsoft's imagehlp.dll.
For a quick look at your PE you can always use Microsoft's dumpbin utility, which is included in Windows SDKs (at least those that ship with a compiler, i.e. those that predate the current 8.1) and many editions of Visual Studio, including the free Community Edition. I also found Daniel Pistelli's free PE Insider quite handy, and the CFF Explorer suite is free as well.
P.S.: probably the best resource documenting the reality (as opposed to the theory) of PE structure is corkami's PE wiki. It is certainly the most comprehensive PE documentation that I've ever seen... ReversingLabs have compiled a document named Undocumented PECOFF which gives an excellent overview over actual requirements vs. the theory. Although their focus is on the security implications, the pdf is also an excellent resource for making PEs work.
