What PE anomalies can crash the Windows Loader or cause a file to not load?

Question

I do malware analysis on Windows. I run hundreds of Windows PEs per day and it is actually relatively common for a file to not run (or sometimes not run on just one specific version of Windows) and I get an error message such as "This is not a valid Win32 application." However, when I open the file in a hex editor, it does have the MZ, PE signature, and even the sections intact. Also, programs such as PE Explorer can open the file fine and claim that it's a valid PE file, without even having to open it in "safemode" for example. Note that I am NOT talking about error messages concerning 64 bit on a 32 bit version of Windows, as that is self-explanatory.

Below is a screenshot of one such file. This file, however, has no DOS stub at all and the DOS header data structures are all set to 0 except for e_lfanew which does point to a PE sig. However, the Windows loader says it's not a valid Win32 app at least on my version of Win7 64bit.

I do know that the loader essentially reads the data structures and from that, it does things such as allocate stack and heap memory, determine which symbols are needed and from which DLL files, as well as a few other tasks. So my assumption based off of that would be that if for example, one of these header data structures told the loader to do something that made no sense such as allocate negative space, too much space, or contained nonsense in a structure that was critical for the loader to work, it could crash it. However, this is just speculation on my part.

Answer 1

Based on my experience, these type of errors could happen when MajorOperatingSystem/MinorOperatingSystem and/or MajorSubsystemVersion/MinorSubsystemVersion are not compatible with the system you are trying to run the PE file. You can find more information about that in Corkami page regarding PE file format. Also, you can look at the Peering inside the PE... paper on Microsoft website which says for these fields:

WORD MajorOperatingSystemVersion WORD MinorOperatingSystemVersion The minimum version of the operating system required to use this executable. This field is somewhat ambiguous since the subsystem fields (a few fields later) appear to serve a similar purpose. This field defaults to 1.0 in all Win32 EXEs to date.

and

WORD MajorSubsystemVersion WORD MinorSubsystemVersion Contains the minimum subsystem version required to run the executable. A typical value for this field is 3.10 (meaning Windows NT 3.1).

Also, if you are interested in anomalies and undocumented stuff of the PE file format, I suggest this very good paper which deals with anomalies in PE which can break poorly implemented parsers. Also, this site of Corkami contains POCs of various working PE samples (both source code and binaries), which test the format to its limits and can be used for testing.

Hope this helps out a bit.

Answer 2

There are many ways in which the loader can fail to load a seemingly valid file. There are differences in rules for 32-bit and 64-bit architectures, for example. The most obvious of those is the minimum file size.

Then there are differences in rules between different versions of Windows. In older versions, critical data such as the import table could be placed in the file header. This is no longer the case.

An alignment requirement for the PE header was introduced. The rules changed for where the PE header can appear in the file. The rules changed for the maximum number of sections in a file.

The rules also changed for section alignment, section order, and overlapping sections, among other things.

The full list of what was changed and how it changed would be very long and detailed, depending on how many versions of Windows are being considered.

Without having your file in hand, it isn't clear why the file won't load.