How to quickly distinguish PE/DLL/DOS-MZ files based on magic numbers?

Question

I know that the specifications of the Microsoft PE/DLL/DOS-MZ files states that the two first bytes of a PE/DLL/DOS-MZ file is MZ (0x4d,0x5a) orZM(0x5a,0x4d`).

The problem with such a small signature is that a lot of other files may match the same specification and a test based only on this two first bytes quickly tends to be inconclusive.

So, my question is simple, after testing that the two first bytes are MZ (or ZM), what other, more reliable, test can be performed to check that the file is a PE/DLL/DOS-MZ?

Answer 1

That would be e_lfanew in the MZ header itself. For simple (16 bit) MSDOS software, this ought to be 0, and for any extended format it should point right to one of the specific LE, PE, or NE headers' identifiers (which all are exactly those two characters). As in all of these cases the header itself is a mere stub, e_cp (the total size of this executable in pages) should not be larger than the offset of e_lfanew. (Plus or minus that infernal e_cblp, the "bytes on last page of file" – I always forget which one it is.)

See Peering inside the PE, an article that despite its age still is very useful.

Answer 2

An old DOS EXE header is only 28 (0x1C) bytes long and is usually followed by the DOS relocation table if present. The IMAGE_DOS_HEADER struct of the NT PE header is much larger at 64 (0x40) bytes as it has been extended for the various other Windows executable formats.

Trying to interpret e_lfanew at offset 60 (0x3C) for a plain DOS executable as suggested by the recommended answer is incorrect as this pulls in whatever data happens to be at that offset, usually from the DOS relocation table but it can vary between valid DOS executables. Using a handful of old DOS executables, the value at this position might not be zero, thus any logic that tries to use this as a distinguishing marker may crash or work incorrectly.

When trying to distinguish a plain DOS EXE, you can't reliably look at any members past e_ovno (overlay number) of the IMAGE_DOS_HEADER struct because they are Windows and OS/2 extensions to the DOS EXE header and do not exist in plain DOS executables.

As far as distinguishing between a DOS executable and a PE executable, I have used the following logic with success:

1. If the beginning of the file does not begin with "MZ" or "ZM", it is not an DOS or Windows executable image. Otherwise you may have one of the following types of executable formats: plain DOS, NE (Windows 16-bit), LE (16-bit VXD), PE32, or PE32+ (PE64).

2. Determine if you have a plain DOS executable by looking at the e_lfanew value. A plain DOS executable will have an out-of-range e_lfanew pointing outside of the limits of the file, a zero, or if the offset happens to be in range, the signature at its offset won't match any signatures below.

3. Try to match the signature of the "in-range" offset pointed to by e_lfanew with the following WORD or DWORD values:

   "PE" followed by two zero bytes if the image is a PE32 or PE32+ (PE64) and is further determined by the "magic" in the NT Optional Header
   "NE" indicates the image is a 16-bit Windows executable
   "LE" indicates the image is a 16-bit Virtual Device Driver (VXD)