8

I am looking at some x86 code, which I believe was built using a Microsoft tool chain, and am trying to figure out the calling convention used during this call:

   push esi ; save ESI (it gets restored later)
   lea esi, [ebp-0xC] ; set param 1 for call to FOO
   call FOO
   test eax, eax ; test return value
   jz somelabel

The function FOO starts like this:

   FOO:
   mov edi, edi
   push ebx
   xor ebx, ebx
   push ebx ; null
   push esi ; pass ESI in as second param to upcoming call, which has been set by caller
   push ptr blah
   mov [esi+0x8], ebx
   mov [esi+0x4], ebx
   mov [esi], ebx
   call InterlockedCompareExchange ; known stdcall func which takes 3 params
   test eax, eax
   ...

as ESI is not initialized in the body of FOO, I have assumed it is passed in as a param by the caller.

What is this calling convention? It looks to be a variant of fastcall. Is there a name for this convention?

  • Isn't it a struct passed through the arguments ? The esi will be the base address and access to each field is passed through several offsets. No ? – perror Aug 22 '13 at 15:47
  • yes the parameter is a pointer to a local variable (12 byte structure) of the caller, but what is the calling convention that passes this pointer as a parameter to FOO? – QAZ Aug 22 '13 at 15:54
12

There is no "official" calling convention that works like that. What you're seeing is most likely the result of Link-time Code Generation, also known as LTO (Link-time optimization) or WPO (Whole program optimization).

When it is enabled, the optimization and code generation is done at link time, when the compiler has access to the code of whole program and all compile units, and can use this information for the more extreme optimizations.

From MSDN:

When /LTCG is used to link modules compiled by using /Og, /O1, /O2, or /Ox, the following optimizations are performed:

  • Cross-module inlining

  • Interprocedural register allocation (64-bit operating systems only)

  • Custom calling convention (x86 only)

  • Small TLS displacement (x86 only)

  • Stack double alignment (x86 only)

  • Improved memory disambiguation (better interference information for global variables and input parameters)

In the code snippet you quoted the compiler detected that the function FOO is not called from outside of the program, so it could customize the calling convention to something that uses register values already set up at the place of call, or otherwise improve register allocation. With heavily templated code you can even get several copies of often-used functions that accept arguments in different sets of registers and/or stack.

  • 2
    What exactly is "stack double alignment"? – newgre Aug 23 '13 at 8:22
  • Don't know for sure, but maybe they mean avoiding having to re-align stack in every function that works with SSE data if they know it's already aligned in every place the function is called. – Igor Skochinsky Aug 23 '13 at 10:59
  • 1
    Not just SSE data, but any data that works better 8-byte aligned (e.g. double primitives). See this valuable post that appears to come from a Microsoft employee for a description of each LTCG optimization. codeproject.com/Articles/1660/… – ajs410 Dec 12 '13 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.