What x86 calling convention passes first parameter via ESI?

Question

I am looking at some x86 code, which I believe was built using a Microsoft tool chain, and am trying to figure out the calling convention used during this call:

   push esi ; save ESI (it gets restored later)
   lea esi, [ebp-0xC] ; set param 1 for call to FOO
   call FOO
   test eax, eax ; test return value
   jz somelabel

The function FOO starts like this:

   FOO:
   mov edi, edi
   push ebx
   xor ebx, ebx
   push ebx ; null
   push esi ; pass ESI in as second param to upcoming call, which has been set by caller
   push ptr blah
   mov [esi+0x8], ebx
   mov [esi+0x4], ebx
   mov [esi], ebx
   call InterlockedCompareExchange ; known stdcall func which takes 3 params
   test eax, eax
   ...

as ESI is not initialized in the body of FOO, I have assumed it is passed in as a param by the caller.

What is this calling convention? It looks to be a variant of fastcall. Is there a name for this convention?

Answer 1

There is no "official" calling convention that works like that. What you're seeing is most likely the result of Link-time Code Generation, also known as LTO (Link-time optimization) or WPO (Whole program optimization).

When it is enabled, the optimization and code generation is done at link time, when the compiler has access to the code of whole program and all compile units, and can use this information for the more extreme optimizations.

From MSDN:

When /LTCG is used to link modules compiled by using /Og, /O1, /O2, or /Ox, the following optimizations are performed:

• Cross-module inlining

• Interprocedural register allocation (64-bit operating systems only)

• Custom calling convention (x86 only)

• Small TLS displacement (x86 only)

• Stack double alignment (x86 only)

• Improved memory disambiguation (better interference information for global variables and input parameters)

In the code snippet you quoted the compiler detected that the function FOO is not called from outside of the program, so it could customize the calling convention to something that uses register values already set up at the place of call, or otherwise improve register allocation. With heavily templated code you can even get several copies of often-used functions that accept arguments in different sets of registers and/or stack.