Questions tagged [calling-conventions]

As assembler languages have no built-in primitives to handle arguments passing while starting a procedure, most of assembler programs and compiler do agree on calling conventions to describe how to emulate arguments passing at the assembler level.

Filter by
Sorted by
Tagged with
1
vote
1answer
42 views

IDA Pro - configure register clobbering

TL;DR: The __fastcall convention in IDA Pro assumes that all the registers which could be used for transferring parameters are clobbered after a call (tested on x86, clobbered regs are eax, edx, ebx, ...
0
votes
0answers
60 views

How to use aarch64 call convention to understand an unknown function?

I’m trying to reverse engineer an aarch64 Golang binary which happens to call a symbol, i.e., function named guess_me which is an import. To stop the binary from throwing errors, I write a Hello World!...
1
vote
0answers
41 views

Weird x86 function calling convention [duplicate]

I have come across some weird function. It's in an x86 LE Windows executable compiled with Visual Studio and linking to v100 runtime. The function seems to be an __stdcall as it ends with a move into ...
0
votes
1answer
117 views

Calling MessageBoxW x64 confused about pointer

OK, I am very new to x64dbg. I used to use OllyDbg a lot and I'm having some problems with this. I just made a customized version of MS-PAINT for practice by changing a few things like the default ...
0
votes
1answer
465 views

8086 Function Calling Conventions

I am reversing an old DOS game from 1989. What are the function calling conventions for the 16-bit 8086 processor that is used to run this type of program? How it works on Windows x64 By function ...
0
votes
1answer
278 views

How to track output parameters in disassembly?

I have the following disassembly of winload!CmpFindNlsData function and I need to check the output parameters of this function. P.S.: CmpFindNLSData(int a1<eax>, int a2, PUNICODE_STRING ...
1
vote
1answer
146 views

Detours not cleaning the stack correctly

I'm trying to hook a function using Detours for the first time. I'm new to reversing software and to hooking, so I may have missed something big here. I'm trying to use this function as a hook: ...
3
votes
1answer
276 views

CreateTimerQueueTimer arguments differs from WinAPI

I was reading a man page about CreateTimerQueueTimer. BOOL CreateTimerQueueTimer( PHANDLE phNewTimer, HANDLE TimerQueue, WAITORTIMERCALLBACK Callback, PVOID ...
1
vote
1answer
341 views

indirect return value address

I have usercall with calling convention I do not fully understand, it returns std::string but IDA recognize it as void. I've noticed that every calle reads from x8/w8 afterwards, from wikipedia; ...
0
votes
1answer
152 views

Why doesn't alloca_probe follow x64 calling convention?

I was practising reverse engineering on some Windows x64 applications when I came across this function: call alloca_probe This alloca_probe function has some strange implementation: EAX is used as ...
1
vote
1answer
151 views

Lack of arguments before a CALL instruction

I'm reversing an open-source windows application written in C++. I found in the disassembler the desired function that I'm trying to understand its behavior. In the code this function is defined as ...
1
vote
2answers
2k views

IDA Hex-Rays: How to SAFELY fix incorrect function declarations?

Update: The problem turned out to be more complex and complex because Hex-Rays incorrectly restores the stack after calls of stdcall functions from cdecl functions: .text:00403F2F 074 ...
2
votes
0answers
406 views

Show parameters stored in stack with __fastcall in IDA

This is an extension of my previous question. Here is a sample of assembly in x64 Windows executable using COM virtual function with __fastcall convention. mov [rsp+32], rdx ...
2
votes
1answer
189 views

What does __return_ptr do in IDA?

Let's say I have this program: struct A { int x; }; struct A func1(int z) { struct A result; result.x = z+1; return result; } int main() { return func1(1).x; } When compiling ...
5
votes
1answer
1k views

How to specify stack-based return values in IDA Pro (GoLang)

tl;dr: I am reversing a GoLang 1.10 executable, compiled for Windows. I am trying to make IDA correctly recognize the calling convention. Details: I am looking at an x86 executable where the ...
2
votes
2answers
2k views

Confused about the value of edx and eax registers used at function entry

I am currently using IDA to disassemble a keygen. In the first few lines of a specific function before the eax and edx registers are given a value, their contents are moved to stack variables var_40 ...
2
votes
2answers
469 views

What compiler uses a calling convention that uses EAX as the first argument, then pushes onto the stack?

I recently asked to question: How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? After posting the question, I found that another function passed the first argument ...
1
vote
1answer
126 views

How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? [duplicate]

A C++ program I'm looking at appears to have been compiled with MSVC. It links to MSVCR100.dll, contains MSVC's security cookie code and uses the Microsoft mangling scheme. However, one of the ...
6
votes
1answer
273 views

Returning value from faux DLL

A bit of background; I'm trying to inject my own code into the old game SimTower with the ultimate goal of reverse engineering it in the same way that OpenRCT2 was created. Since SimTower is ...
8
votes
2answers
1k views

Register Calling Convention: written in stone, or in mud?

When disassembling an old Delphi 3 executable, I find some routines that pass arguments in registers EAX, EDX, and on the stack – but not in ECX! For those routines, ECX never gets set to a '...
4
votes
1answer
1k views

How to find the stack and other info in a memory dump? ARM

I'm reversing a closed platform to try gain execution using an exploit, a stack overflow. I've been told that (since there are no debuggers) the best way is to use RAM dumps (that I have) to try ...
1
vote
0answers
362 views

IDA Pro .til files backward compatibility

I read in the IDA Pro Book that you may use till.exe to parse header files and create standalone .til files which can be made globally available by copying them into <IDADIR>/til. This ...
2
votes
1answer
556 views

How can I call a function that acts like __thiscall, except the caller cleans the stack?

I'm trying to call a function with a function pointer in C++, but I can't find any convention that fits it. Its stack arguments are (right to left): func(float x, float y, int unk); ...but ecx needs ...
1
vote
2answers
1k views

Confused about calling convention in a Windows kernel mode driver

Reverse engineering a kernel mode driver (in its 32-bit x86 incarnation) I stumbled over what seems to be an odd calling convention. For a driver I'd expect to see __cdecl, __fastcall and __stdcall in ...
1
vote
1answer
683 views

Call a thiscall method from an injected DLL [closed]

I am trying to create a (32-bit) DLL to inject into a target process. My DLL should call a function inside the original process. So far nothing special, however the target function is using the ...
1
vote
1answer
149 views

Calling a function outside the module

Is it possible that a module will have a call to a function in a different module? I know about import tables and delayed import tables. But are there any more options to do so? How will it look in ...
3
votes
1answer
2k views

Why is this _DllMainCRTStartup calling convention not as expected?

I've been looking around at various DLLs on my computer lately and I thought a decent place to start was the image entry point (typically _DllMainCRTStartup). From what I can see of the CRT source ...
3
votes
1answer
3k views

How to create function prototype that recognizes arguments passed by reference in IDA Pro

I have a function prototype generated by IDA that looks like this: However, it looks like there is a struct being referenced at the instruction lea edi, [esp+290h+var_240]. I would like IDA to ...
3
votes
2answers
868 views

x86_64 Calling convention

I am slightly confused by some x86_64 linux code here (generated by llvm): # BB#0: # %entry push RBP .Ltmp21: .cfi_def_cfa_offset 16 .Ltmp22: ....
2
votes
2answers
977 views

How to programmatically identify binary calling convention?

I have a problem where I'm instrumenting stripped binaries; I don't know the start of main(). But there's always an init(), and init() calls libc_start_main(), which receives a pointer to main. If I ...
2
votes
2answers
1k views

Change calling convention from IDAPython?

I have a 64-bit Mach-O sample that uses the System V AMD64 ABI calling convention (rdi, rsi, rdx, rcx,...); but IDA thinks all the functions are using cdecl. I'm new to IDA scripting but didn't find ...
13
votes
3answers
9k views

Which calling convention to use for EAX/EDX in IDA

I have some code (I assume Delphi) which uses only the EAX and EDX register for passing the arguments (and of course the stack if more are required). I looked which calling conventions would match, ...
8
votes
1answer
3k views

What x86 calling convention passes first parameter via ESI?

I am looking at some x86 code, which I believe was built using a Microsoft tool chain, and am trying to figure out the calling convention used during this call: push esi ; save ESI (it gets ...
6
votes
3answers
8k views

IDA Pro function stack frame view

IDA Pro displays certain buffer or padding above (at lower addresses) local variables in stack frame view. For instance: Example 1. The following screen shot of stack frame view shows 12 bytes (...
8
votes
2answers
2k views

Identifying variable args function

How would a C variable argument function such as printf(char* format, ...) look like when disassembled? Is it always identified by calling convention, or are there more ways to identify it?