Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [calling-conventions]

As assembler languages have no built-in primitives to handle arguments passing while starting a procedure, most of assembler programs and compiler do agree on calling conventions to describe how to emulate arguments passing at the assembler level.

1
vote
2answers
125 views

IDA Hex-Rays: How to SAFELY fix incorrect function declarations?

Update: The problem turned out to be more complex and complex because Hex-Rays incorrectly restores the stack after calls of stdcall functions from cdecl functions: .text:00403F2F 074 ...
2
votes
0answers
96 views

Show parameters stored in stack with __fastcall in IDA

This is an extension of my previous question. Here is a sample of assembly in x64 Windows executable using COM virtual function with __fastcall convention. mov [rsp+32], rdx ...
2
votes
1answer
79 views

What does __return_ptr do in IDA?

Let's say I have this program: struct A { int x; }; struct A func1(int z) { struct A result; result.x = z+1; return result; } int main() { return func1(1).x; } When compiling ...
4
votes
1answer
229 views

How to specify stack-based return values in IDA Pro (GoLang)

tl;dr: I am reversing a GoLang 1.10 executable, compiled for Windows. I am trying to make IDA correctly recognize the calling convention. Details: I am looking at an x86 executable where the ...
2
votes
2answers
686 views

Confused about the value of edx and eax registers used at function entry

I am currently using IDA to disassemble a keygen. In the first few lines of a specific function before the eax and edx registers are given a value, their contents are moved to stack variables var_40 ...
2
votes
2answers
210 views

What compiler uses a calling convention that uses EAX as the first argument, then pushes onto the stack?

I recently asked to question: How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? After posting the question, I found that another function passed the first argument ...
1
vote
1answer
92 views

How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? [duplicate]

A C++ program I'm looking at appears to have been compiled with MSVC. It links to MSVCR100.dll, contains MSVC's security cookie code and uses the Microsoft mangling scheme. However, one of the ...
6
votes
1answer
191 views

Returning value from faux DLL

A bit of background; I'm trying to inject my own code into the old game SimTower with the ultimate goal of reverse engineering it in the same way that OpenRCT2 was created. Since SimTower is ...
6
votes
2answers
684 views

Register Calling Convention: written in stone, or in mud?

When disassembling an old Delphi 3 executable, I find some routines that pass arguments in registers EAX, EDX, and on the stack – but not in ECX! For those routines, ECX never gets set to a '...
4
votes
1answer
716 views

How to find the stack and other info in a memory dump? ARM

I'm reversing a closed platform to try gain execution using an exploit, a stack overflow. I've been told that (since there are no debuggers) the best way is to use RAM dumps (that I have) to try ...
1
vote
0answers
190 views

IDA Pro .til files backward compatibility

I read in the IDA Pro Book that you may use till.exe to parse header files and create standalone .til files which can be made globally available by copying them into <IDADIR>/til. This ...
2
votes
1answer
329 views

How can I call a function that acts like __thiscall, except the caller cleans the stack?

I'm trying to call a function with a function pointer in C++, but I can't find any convention that fits it. Its stack arguments are (right to left): func(float x, float y, int unk); ...but ecx needs ...
1
vote
2answers
534 views

Confused about calling convention in a Windows kernel mode driver

Reverse engineering a kernel mode driver (in its 32-bit x86 incarnation) I stumbled over what seems to be an odd calling convention. For a driver I'd expect to see __cdecl, __fastcall and __stdcall in ...
1
vote
1answer
440 views

Call a thiscall method from an injected DLL [closed]

I am trying to create a (32-bit) DLL to inject into a target process. My DLL should call a function inside the original process. So far nothing special, however the target function is using the ...
2
votes
1answer
102 views

Calling a function outside the module

Is it possible that a module will have a call to a function in a different module? I know about import tables and delayed import tables. But are there any more options to do so? How will it look in ...
3
votes
1answer
731 views

Why is this _DllMainCRTStartup calling convention not as expected?

I've been looking around at various DLLs on my computer lately and I thought a decent place to start was the image entry point (typically _DllMainCRTStartup). From what I can see of the CRT source ...
3
votes
1answer
2k views

How to create function prototype that recognizes arguments passed by reference in IDA Pro

I have a function prototype generated by IDA that looks like this: However, it looks like there is a struct being referenced at the instruction lea edi, [esp+290h+var_240]. I would like IDA to ...
3
votes
2answers
688 views

x86_64 Calling convention

I am slightly confused by some x86_64 linux code here (generated by llvm): # BB#0: # %entry push RBP .Ltmp21: .cfi_def_cfa_offset 16 .Ltmp22: ....
2
votes
2answers
499 views

How to programmatically identify binary calling convention?

I have a problem where I'm instrumenting stripped binaries; I don't know the start of main(). But there's always an init(), and init() calls libc_start_main(), which receives a pointer to main. If I ...
2
votes
2answers
756 views

Change calling convention from IDAPython?

I have a 64-bit Mach-O sample that uses the System V AMD64 ABI calling convention (rdi, rsi, rdx, rcx,...); but IDA thinks all the functions are using cdecl. I'm new to IDA scripting but didn't find ...
12
votes
3answers
6k views

Which calling convention to use for EAX/EDX in IDA

I have some code (I assume Delphi) which uses only the EAX and EDX register for passing the arguments (and of course the stack if more are required). I looked which calling conventions would match, ...
8
votes
1answer
2k views

What x86 calling convention passes first parameter via ESI?

I am looking at some x86 code, which I believe was built using a Microsoft tool chain, and am trying to figure out the calling convention used during this call: push esi ; save ESI (it gets ...
6
votes
3answers
6k views

IDA Pro function stack frame view

IDA Pro displays certain buffer or padding above (at lower addresses) local variables in stack frame view. For instance: Example 1. The following screen shot of stack frame view shows 12 bytes (...
8
votes
2answers
1k views

Identifying variable args function

How would a C variable argument function such as printf(char* format, ...) look like when disassembled? Is it always identified by calling convention, or are there more ways to identify it?