What are indicators for hitting the OEP?

Question

for me, it is sometimes difficult to decide if I have hit the OEP when I try to manuelly unpack it. Especially in cases, when it does not begin with the standard function prologue.

For example: I have found a following sample which starts with a PUSHAD instruction. I step through the content of code until I reach POPAD. After few lines, I have seen a JMP intruction which leads to a CALL instruction and a function is called. After stepping over that function, in the next line, there is another JMP instruction which ends up here:

    PUSH 58
    PUSH unbr002.014A22F8
    CALL unbr002.013CBD40#
    XOR ESI, ESI
    MOV DWORD PTR SS:[EBP-4], ESI
    LEA EAX, DWORD PTR SS:[EBP-68]
    PUSH EAX
    CALL DWORD PTR DS:[141409C]          kernel32.GetStartupInfoA
    ....
    ....
    ....
    ....
    POP ECX
    CALL DWORD PTR DS:[1414278]          kernel32.GetCommandLineA
    ....

So, I have read somewhere that "GetCommandLineA – indicating you’ve hit the entrypoint of a program compiled with visual studio 6"

and I asked myself if there is a list of indicators to know if one has hit the OEP.

best regards,

Answer 1

Magic constants can sometimes help even if the packer splits the startup code into tiny fragments with copious jumping around and 'returning' to pushed/swapped addresses and so on.

One such example is __security_init_cookie for VC++ startup code, which is related to the constant 0BB40E64Fh in 32-bit code and 2B992DDFA232h for 64-bit. That function is normally the only code which writes to both __security_cookie and __security_cookie_complement:

.text:00401F06 020           mov     edi, 0BB40E64Eh
; ... stuff involving GetCurrentThreadId(), GetCurrentProcessId() etc. pp. ...
.text:00401F79 020           mov     ___security_cookie, ecx
.text:00401F7F 020           not     ecx
.text:00401F81 020           mov     ___security_cookie_complement, ecx
...
.data:00404000     ___security_cookie dd 0BB40E64Eh
.data:00404004     ___security_cookie_complement dd 44BF19B1h

The location of ___security_cookie is also easy to identify because of its unique role, independent of magic constants. And it leads you straight to __security_init_cookie() which is often the first function called at OEP, even before __tmainCRTStartup(). By contrast, imported functions (CRT dll, Windows API) are occasionally scrambled and thus difficult to identify.

There's some crapware out there that automatically obfuscates constant loads by splitting them like this:

1441285DB           mov     edx, 0E6FFA20Fh                           
...
1441285E3           lea     edx, [rdx+19005DF2h]                      

In this example the obfuscated constant is 1, of course. Other splits are possible, e.g. involving arithmetic or bit ops instead of LEA. The signature property is a constant modifying a constant residing in a register (inside a basic block, without the involvement of relocations), something which all self-respecting compilers would optimise away.