I recently asked to question: How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? After posting the question, I found that another function passed the first argument in EAX and then pushed its remaining argument. The caller then cleans up the stack.

The calling code:

.text:00402465                 lea     eax, [ebp+var_4]
.text:00402469                 push    eax
.text:0040246A                 mov     eax, [ebp+hWnd]
.text:0040246D                 call    openFileDialog
.text:00402472                 add     esp, 4

And the function itself:

.text:00411730 openFileDialog  proc near
.text:00411730 arg_0           = dword ptr  8
.text:00411730                 push    ebp
.text:00411731                 mov     ebp, esp
.text:00411733                 sub     esp, 18h
.text:00411736                 cmp     byte_42AE1D, FALSE
.text:0041173D                 push    ebx
.text:0041173E                 push    esi
.text:0041173F                 push    edi
.text:00411740                 mov     esi, eax
.text:00411789                 mov     eax, [ebp+arg_0]
.text:0041178C                 push    eax
.text:0041178D                 push    esi
.text:0041178E                 call    openFileDialog_Compat
.text:00411793                 add     esp, 8

As you can see, in the function, the value of EAX is saved before anything can affect it, so it is definitely being used as a parameter. Later, the pushed argument is passed to a normal __cdecl function.

The program is linked to use msvcr100.dll and uses MSVC style throughout (Such as __security_cookie, MSVC name mangling, etc.), so it would appear to have been compiled with Visual C++, but this unusual calling convention makes me question that.

  • 3
    When using "Whole Program Optimization", the compiler may use whatever calling convention fits for functions with internal linkage. – newgre Dec 6 '17 at 8:10

This is probably a program compiled with "Whole Program Optimization" or "Link-time code generation". From MSDN:

When /LTCG is used to link modules compiled with /Og, /O1, /O2, or /Ox, the following optimizations are performed:

  • Cross-module inlining
  • Interprocedural register allocation (64-bit operating systems only)

  • Custom calling convention (x86 only)

  • Small TLS displacement (x86 only)

  • Stack double alignment (x86 only)

  • Improved memory disambiguation (better interference information for global variables and input parameters)

  • Thank you. This is something I have not encountered before. I've only ever thought from the compiler point of view and the described standards. – Ben Jaguar Marshall Dec 7 '17 at 1:10

check Calling Conventions that msvc support yet. as for the function name openFileDialog, it's a c# func? combining C# and C++ via CLR?? strange things may happen. MS dose not need to expose these details to public.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.