What compiler uses a calling convention that uses EAX as the first argument, then pushes onto the stack?

Question

I recently asked to question: How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? After posting the question, I found that another function passed the first argument in EAX and then pushed its remaining argument. The caller then cleans up the stack.

The calling code:

.text:00402465                 lea     eax, [ebp+var_4]
    ...
.text:00402469                 push    eax
.text:0040246A                 mov     eax, [ebp+hWnd]
.text:0040246D                 call    openFileDialog
.text:00402472                 add     esp, 4

And the function itself:

.text:00411730 openFileDialog  proc near
.text:00411730
    ...
.text:00411730 arg_0           = dword ptr  8
.text:00411730
.text:00411730                 push    ebp
.text:00411731                 mov     ebp, esp
.text:00411733                 sub     esp, 18h
.text:00411736                 cmp     byte_42AE1D, FALSE
.text:0041173D                 push    ebx
.text:0041173E                 push    esi
.text:0041173F                 push    edi
.text:00411740                 mov     esi, eax
    ...
.text:00411789                 mov     eax, [ebp+arg_0]
.text:0041178C                 push    eax
.text:0041178D                 push    esi
.text:0041178E                 call    openFileDialog_Compat
.text:00411793                 add     esp, 8

As you can see, in the function, the value of EAX is saved before anything can affect it, so it is definitely being used as a parameter. Later, the pushed argument is passed to a normal __cdecl function.

The program is linked to use msvcr100.dll and uses MSVC style throughout (Such as __security_cookie, MSVC name mangling, etc.), so it would appear to have been compiled with Visual C++, but this unusual calling convention makes me question that.

Answer 1

This is probably a program compiled with "Whole Program Optimization" or "Link-time code generation". From MSDN:

When /LTCG is used to link modules compiled with /Og, /O1, /O2, or /Ox, the following optimizations are performed:

• Cross-module inlining

• Interprocedural register allocation (64-bit operating systems only)

• Custom calling convention (x86 only)

• Small TLS displacement (x86 only)

• Stack double alignment (x86 only)

• Improved memory disambiguation (better interference information for global variables and input parameters)

Answer 2

check Calling Conventions that msvc support yet. as for the function name openFileDialog, it's a c# func? combining C# and C++ via CLR?? strange things may happen. MS dose not need to expose these details to public.