3

I've been looking around at various DLLs on my computer lately and I thought a decent place to start was the image entry point (typically _DllMainCRTStartup).

From what I can see of the CRT source code (MSVC) this calls another function (__DllMainCRTStartup) with the same arguments (HMODULE, DWORD, LPVOID). However many of the binaries I've looked at seem to deviate from this:

  push        dword ptr [ebp+8]
  mov         ecx,dword ptr [ebp+10h]
  mov         edx,dword ptr [ebp+0Ch]
  call        10024E5B // __DllMainCRTStartup

This looks to be pushing on the first parameter (HMODULE) and then passing the other two via registers ecx/edx. The function being called expects them to be there as well. What I expected to see was all three being pushed on the stack which I believe is how the _cdecl convention works. This is an example:

  push        dword ptr [ebp+10h]
  push        dword ptr [ebp+0Ch]
  push        dword ptr [ebp+8]
  call        3DC40E2E // __DllMainCRTStartup

Does anyone know where the first variant is coming from? Is it an older version of the CRT? Or maybe a different compiler entirely? Bonus if you know what kind of calling convention it is as well, looks like a strange cousin of __fastcall to me!

Thanks very much!

Edit:

Thought I'd add the function prototype from the CRT source code (the version that comes with VS 2012 at least)

__declspec(noinline)
BOOL __cdecl
__DllMainCRTStartup(
    HANDLE  hDllHandle,
    DWORD   dwReason,
    LPVOID  lpreserved
    )

Editx2:

I've just seen the 'Related' link: What x86 calling convention passes first parameter via ESI?

It seems this might be a result of LTCG and the linker basically just deciding to do whatever it likes because it knows about all the components!

2
  • Why should __DllMainCRTStartup adhere to any calling convention? noone is forcing the compiler to do so - it's not a public symbol that can be called from external modules anyways.
    – newgre
    Feb 5, 2016 at 19:09
  • Well it doesn't have to obviously, I was just wondering why it wasn't a __cdecl when it's marked as such in the source
    – B A
    Feb 5, 2016 at 19:28

1 Answer 1

5

With global optimisation (link-time code generation) the compiler is free to ignore the declared calling convention for all functions that aren't externally visible - that is, neither exported nor otherwise pinned by having their addresses taken and passed to external code.

This can make the resulting binaries difficult to analyse since even the well-known parts of the startup code can morph and fuse under global optimisation. It is still recognisable by its behaviour and embedded magic constants (like those for initialising the stack guard cookie and its complement) but the success rate of library object code recognition by simple mechanisms like IDA's FLIRT takes a nose dive.

Also, in this situation the function prototypes pulled from headers or from IDA's signature files may be wrong as likely as not. This can lead IDA's analysis engine (and the reader) astray and thus cause problems even several call levels away from the problematic functions.

In short: with LTCG all bets are off and simple analysis code can become unreliable.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.