7 votes
Accepted

What compiler uses a calling convention that uses EAX as the first argument, then pushes onto the stack?

This is probably a program compiled with "Whole Program Optimization" or "Link-time code generation". From MSDN: When /LTCG is used to link modules compiled with /Og, /O1, /O2, or /Ox, the ...
  • 35.1k
7 votes
Accepted

Register Calling Convention: written in stone, or in mud?

If the compiler can prove that it has all call sites for a given function under its control then it can discard conventions and arrange things around to its liking. Microsoft's C/C++ compiler has been ...
  • 1,960
7 votes
Accepted

How to create function prototype that recognizes arguments passed by reference in IDA Pro

My guess is that the function signature should be something like: int __usercall Call_HTTP@<eax>(int x, int y, void* http_object@<edi>); __usercall means the calling convention for the ...
  • 402
6 votes
Accepted

Returning value from faux DLL

Your faux function is not fixing up the stack on the epilog of the function. It should have pop ds pop bp dec bp at the end by the calling convention. If you look at the original function it fixes ...
  • 1,331
5 votes
Accepted

How to find the stack and other info in a memory dump? ARM

The only idea I have is to compare the dumps. The places that are same in all dumps are code or read only data. The places that are changing from dump to dump are either stack or section like .bss. ...
  • 7,945
5 votes

Why is this _DllMainCRTStartup calling convention not as expected?

With global optimisation (link-time code generation) the compiler is free to ignore the declared calling convention for all functions that aren't externally visible - that is, neither exported nor ...
  • 1,960
3 votes

8086 Function Calling Conventions

Calling conventions are compiler specific, meaning they are specified by the compiler manufacturer like Microsoft, and not by the 8086 manufacturer like Intel. In case you program in Assembler or ...
  • 1,814
3 votes
Accepted

What does __return_ptr do in IDA?

The "problem" with your example is that the structure is too small (four bytes), so it fits in a register and is not actually passed on the stack. From the Itanium C++ ABI (used by most GCC ...
  • 35.1k
3 votes

x86_64 Calling convention

So, your question is mainly about this part of the code (if I do not care about the second call which is similar anyway): mov EDI, .L.urandom mov ESI, 8 call open In fact, this is perfectly okay ...
  • 18.1k
3 votes
Accepted

How to programmatically identify binary calling convention?

Since you mention __libc_start_main it seems you need to worry only about Linux. In such case, the calling convention is known and fixed. In most cases it will be SystemV ABI or similar.
  • 35.1k
2 votes
Accepted

How can I call a function that acts like __thiscall, except the caller cleans the stack?

Sounds like you have a variadic __thiscall function (printf-like). Those use ecx for this but the remaining arguments are pushed onto the stack and the caller cleans it up.
  • 35.1k
2 votes
Accepted

How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX?

We can see from this list that IA-32's Delphi/Free Pascal calling convention is the register calling convention. My guess is you're dealing with a binary that's the result of something like this: How ...
  • 2,164
2 votes
Accepted

CreateTimerQueueTimer arguments differs from WinAPI

Yep, I think I've seen a similar case before. It happens very rarely, but because you can do some stunts at the C preprocessor level as well as at the linker level, this happens to work. In this case ...
  • 10.5k
2 votes
Accepted

indirect return value address

From https://github.com/Siguza/ios-resources/blob/master/bits/arm64.md x8 pointer to where to write the return value if >128 bits, otherwise scratch register So it seems x8 is used to pass an ...
2 votes

IDA Hex-Rays: How to SAFELY fix incorrect function declarations?

The decompiler decided that ecx is used by sub_431C00 because of push ecx at the beginning of the function which fills the stack slot later used by the variable phkResult, so it may look as if the ...
  • 35.1k
2 votes
Accepted

IDA Hex-Rays: How to SAFELY fix incorrect function declarations?

Welcome to RE.SE! How to understand what the problem is? When the function is decompiled, IDA checks for its dependencies via dataflow analysis. Basically, it asks which values have to be defined ...
  • 2,836
1 vote

How to call and what means: _DWORD *__thiscall

Basically I've checked with Ghidra to compare and I've found that the signature is different: void __thiscall FUN_00142860(int param_1_00,undefined4 *param_1,int param_2) So, basically I just ...
  • 131
1 vote

Calling function with string as parameter

At the end I just had to compile as Release. Thanks to this information: https://stackoverflow.com/questions/10315041/meaning-of-acronym-sso-in-the-context-of-stdstring/ So it's just a compiler ...
  • 131
1 vote

Calling MessageBoxW x64 confused about pointer

First make sure you are using correct calling convention, in 64-bit Windows most APIs expect parameters via registers not push. An example of calling convention for 64-bit Windows: lea r9d, uType ...
1 vote

Detours not cleaning the stack correctly

I have found the solution to this problem, which is quite simple. I got confused with the hooked function signature and the __fastcall trick to get a non member function to work with the __thiscall ...
  • 187
1 vote
Accepted

Why doesn't alloca_probe follow x64 calling convention?

This is not a standard function but a compiler helper, used by the compiler to perform some necessary housekeeping (allocate extra stack space) so it doesn’t have to follow the ABI for “normal” ...
  • 35.1k
1 vote

Lack of arguments before a CALL instruction

Seems like it has 4 arguments, and they are passed through stack, yet the more lines of disassembly above E42D6A needed to make sure. Are those mov and lea instruction confusing you? If you've ...
  • 1,770
1 vote
Accepted

How to specify stack-based return values in IDA Pro (GoLang)

I have been in touch with IDA support and unfortunately, right now, there does not seem to be a good way to do this. IDA assumes in many cases that the return value of a function must be stored in a ...
  • 61
1 vote
Accepted

Confused about calling convention in a Windows kernel mode driver

Most likely it's just LTCG/LTO, especially if the function in question is only never called externally. Using it can result in the following: Cross-module inlining Interprocedural register allocation ...
  • 35.1k
1 vote

Confused about calling convention in a Windows kernel mode driver

Many non-Microsoft compilers pass the first argument in EAX. From https://en.wikipedia.org/wiki/X86_calling_conventions: optlink The three lexically first (leftmost) arguments are passed in ...
1 vote

Call a thiscall method from an injected DLL

What error you get? Try something like this: class HashClass { public: byte data[1024]; LPVOID HashFunc(float, char, char); }; typedef LPVOID (HashClass::*HASHFUNC)(float, char, char); ...
  • 304

Only top scored, non community-wiki answers of a minimum length are eligible