22

you can add a function type by editing it (Key Y) and adding the name. I wrote a page to remind me about calling conventions at ASM level. Introduction the original call is myfunc(0,1,2,3,4). standard order is first argument is pushed last. standard stack adjusting is 'callee cleanup' - after returning, the stack should be without its calling arguments. ...


18

It is very simple in some architectures, and not very obvious in others. I'll describe a few I'm familiar with. SystemV x86_64 (Linux, OS X, BSD) Probably the easiest to recognize. Because of the boneheaded decision to specify the number of used XMM registers in al, most vararg functions begin like this: push rbp mov rbp, rsp sub ...


12

There is no "official" calling convention that works like that. What you're seeing is most likely the result of Link-time Code Generation, also known as LTO (Link-time optimization) or WPO (Whole program optimization). When it is enabled, the optimization and code generation is done at link time, when the compiler has access to the code of whole program and ...


10

(My answer is x86-specific). Internally to the function, it looks just like any other function. The only difference being, at some point during the function, it will take the (stack) address of the last non-variable argument, and increment it by the word size on the platform; this is then used as a pointer to the base of the variable arguments. ...


10

If you run into a calling convention which is not covered by any of the standard calling conventions you can use the __usercall or __userpurge calling convention which allows you to specify which arguments are passed where. The syntax is return_type __usercall function_name<registers>(arg0_type arg0<registers>, arg1_type arg1<registers>, ....


7

My guess is that the function signature should be something like: int __usercall Call_HTTP@<eax>(int x, int y, void* http_object@<edi>); __usercall means the calling convention for the function is not a standard one (like stdcall, cdecl, etc.) as the function passes two arguments on stack and one in edi. @<eax> : function returns a value ...


7

This is probably a program compiled with "Whole Program Optimization" or "Link-time code generation". From MSDN: When /LTCG is used to link modules compiled with /Og, /O1, /O2, or /Ox, the following optimizations are performed: Cross-module inlining Interprocedural register allocation (64-bit operating systems only) Custom calling ...


6

Your faux function is not fixing up the stack on the epilog of the function. It should have pop ds pop bp dec bp at the end by the calling convention. If you look at the original function it fixes up the stack correctly. Let's say the stack pointer had the value of 1000h coming into the function (after CS:IP is already pushed on the stack for the far ...


6

If the compiler can prove that it has all call sites for a given function under its control then it can discard conventions and arrange things around to its liking. Microsoft's C/C++ compiler has been doing this for decades in connection with link-time code generation and profile-guided optimisation, especially internal copies of the compiler like the one ...


5

With global optimisation (link-time code generation) the compiler is free to ignore the declared calling convention for all functions that aren't externally visible - that is, neither exported nor otherwise pinned by having their addresses taken and passed to external code. This can make the resulting binaries difficult to analyse since even the well-known ...


5

Delphi and Borland C++ Builder use EAX, EDX and ECX for the first three arguments in their variant of the __fastcall calling convention. So if you choose "Delphi" or "C++ Builder" in Options-Compiler, you can just use __fastcall in the function prototype - no need to resort to __usercall.


5

The only idea I have is to compare the dumps. The places that are same in all dumps are code or read only data. The places that are changing from dump to dump are either stack or section like .bss. After finding places that are not changing I'd try to disassemble these places in order to divide between code and data. I think that the places with the code ...


4

IDA keeps track of the value of the stack pointer (ESP) throughout its static analysis of the entire function. The greatest negative value of ESP (relative to the beginning of the function) is used to determine the size of the stack frame. As for why the stack frames you posted have "undefined" bytes at the top, it's because IDA couldn't automatically ...


3

So, your question is mainly about this part of the code (if I do not care about the second call which is similar anyway): mov EDI, .L.urandom mov ESI, 8 call open In fact, this is perfectly okay to go through ESI and EDI as it is done here, because the value that are transfered are 32 bits values... and, anyway, in amd64, when manipulating 32 bits ...


3

Since you mention __libc_start_main it seems you need to worry only about Linux. In such case, the calling convention is known and fixed. In most cases it will be SystemV ABI or similar.


3

Use idc.SetType (see documentation here ) It works exactly the same as pressing y on function and defining the type, but you have to add semicolon to the end of the type definition.


3

Thank you all so much for your answers and comments. While reading your comments and preparing to update my question, I found the answer. I must give credit to Igor Skochinsky, who asked me to provide functions' prolog instructions. Both functions use the cdecl calling convention. However, calling convention has nothing to do with this buffer. This is what ...


2

The screenshots shows that u are examining IDA's detailed stack view. IDA gives a name to every byte that is accessed directly in the function, any other bytes remain undefined. Calling convention ? Give us the prologue and the epilogue of this subroutine so we can see how the stack is allocated and cleaned. So if this is a stack frame of a normal ...


2

Sounds like you have a variadic __thiscall function (printf-like). Those use ecx for this but the remaining arguments are pushed onto the stack and the caller cleans it up.


2

We can see from this list that IA-32's Delphi/Free Pascal calling convention is the register calling convention. My guess is you're dealing with a binary that's the result of something like this: How to call a function using Delphi's register calling conventions from Visual C++? To partially quote the top-voted answer: Delphi's register calling ...


2

Yep, I think I've seen a similar case before. It happens very rarely, but because you can do some stunts at the C preprocessor level as well as at the linker level, this happens to work. In this case I looked it up in the 3790.1830 DDK (Windows 2003 Server) and several newer WDKs and SDKs. Windows 2000/XP target (3790.1830/inc/w2k/winbase.h and 3790.1830/...


2

The "problem" with your example is that the structure is too small (four bytes), so it fits in a register and is not actually passed on the stack. From the Itanium C++ ABI (used by most GCC implementations): A type is considered non-trivial for the purposes of calls if: it has a non-trivial copy constructor, move constructor, or destructor, or ...


1

I have been in touch with IDA support and unfortunately, right now, there does not seem to be a good way to do this. IDA assumes in many cases that the return value of a function must be stored in a register. This is also the reason for the crash when trying to overrule this inherent assumption with scattered arguments.


1

From https://github.com/Siguza/ios-resources/blob/master/bits/arm64.md x8 pointer to where to write the return value if >128 bits, otherwise scratch register So it seems x8 is used to pass an address at which the return value will land if it's too big, rather than the return value directly, hence indirect I suppose. I couldn't find official ...


1

The decompiler decided that ecx is used by sub_431C00 because of push ecx at the beginning of the function which fills the stack slot later used by the variable phkResult, so it may look as if the initial value of phkResult is taken from ecx: .text:00431C00 push ecx .text:00431C01 lea eax, [esp+4+phkResult] .text:...


1

Welcome to RE.SE! How to understand what the problem is? When the function is decompiled, IDA checks for its dependencies via dataflow analysis. Basically, it asks which values have to be defined in order to make this function work. When it encounters the first line of the function (push ecx) IDA noticed that the value of ecx is used before it was ...


1

There is no need to do anything. For x64 Mach-O and ELF files, cdecl is the same as fastcall and means SysV AMD64 ABI.


1

Most likely it's just LTCG/LTO, especially if the function in question is only never called externally. Using it can result in the following: Cross-module inlining Interprocedural register allocation (64-bit operating systems only) Custom calling convention (x86 only) Small TLS displacement (x86 only) Stack double alignment (x86 only) Improved memory ...


1

Many non-Microsoft compilers pass the first argument in EAX. From https://en.wikipedia.org/wiki/X86_calling_conventions: optlink The three lexically first (leftmost) arguments are passed in EAX, EDX, and ECX... ... Borland register Evaluating arguments from left to right, it passes three arguments via EAX, EDX, ECX. ... Watcom register ...


1

What error you get? Try something like this: class HashClass { public: byte data[1024]; LPVOID HashFunc(float, char, char); }; typedef LPVOID (HashClass::*HASHFUNC)(float, char, char); HASHFUNC *HashFunc = (HASHFUNC *) 0xD34DB33F; void CallHashFunction() { HashClass *hashClass = new HashClass(); (*hashClass.*HashFunc)(1.0f, 'a', 'b'); ...


Only top voted, non community-wiki answers of a minimum length are eligible