Questions tagged [calling-conventions]
As assembler languages have no built-in primitives to handle arguments passing while starting a procedure, most of assembler programs and compiler do agree on calling conventions to describe how to emulate arguments passing at the assembler level.
39
questions
0
votes
0
answers
44
views
Using __usercall convention in IDA with non-x86 architecture
I'm working on SPARC binary in IDA Pro and need to use __usercall convention, but can't make it work. For example, I've tried to use the following function declaration to reflect the fact that the ...
2
votes
0
answers
68
views
Tracing odd variant of call analysis failed
Disclaimer before I get flogged: I'm playing with a commercial app, for which I have a license, for my own education because building toy C apps has run out of challenge (and I know whats inside).
...
- 215
2
votes
1
answer
658
views
What is a "far call" in an x86 or x86_64 cpu
I do not understand the usefulness of the "far call" instruction in a 86 CPU.
On a 32 bits CPU for example each process has an addressing space of 4Gb (0x00000000 to 0xFFFFFFFF).
There can ...
- 797
-1
votes
1
answer
650
views
How to call and what means: _DWORD *__thiscall
I have one question maybe you can help, I always find this functions in IDA:
void __thiscall sub_79340(_BYTE *this, int a2, int a3, int a4)
And I use them like this:
// talkChannel
typedef DWORD(...
- 161
1
vote
1
answer
197
views
Calling function with string as parameter
I'm trying to call a function in a game that makes my character sending a message.
I already called a few functions in this game but this one has a tricky parameter.
The function signature should be ...
- 161
1
vote
1
answer
588
views
IDA Pro - configure register clobbering
TL;DR:
The __fastcall convention in IDA Pro assumes that all the registers which could be used for transferring parameters are clobbered after a call (tested on x86, clobbered regs are eax, edx, ebx, ...
- 13
1
vote
0
answers
46
views
Weird x86 function calling convention [duplicate]
I have come across some weird function. It's in an x86 LE Windows executable compiled with Visual Studio and linking to v100 runtime.
The function seems to be an __stdcall as it ends with a move into ...
- 111
0
votes
1
answer
303
views
Calling MessageBoxW x64 confused about pointer
OK, I am very new to x64dbg. I used to use OllyDbg a lot and I'm having some problems with this.
I just made a customized version of MS-PAINT for practice by changing a few things like the default ...
- 101
2
votes
1
answer
2k
views
8086 Function Calling Conventions
I am reversing an old DOS game from 1989. What are the function calling conventions for the 16-bit 8086 processor that is used to run this type of program?
How it works on Windows x64
By function ...
0
votes
1
answer
364
views
How to track output parameters in disassembly?
I have the following disassembly of winload!CmpFindNlsData function and I need to check the output parameters of this function.
P.S.: CmpFindNLSData(int a1<eax>, int a2, PUNICODE_STRING ...
- 17
1
vote
1
answer
337
views
Detours not cleaning the stack correctly
I'm trying to hook a function using Detours for the first time. I'm new to reversing software and to hooking, so I may have missed something big here.
I'm trying to use this function as a hook:
...
- 187
3
votes
1
answer
334
views
CreateTimerQueueTimer arguments differs from WinAPI
I was reading a man page about CreateTimerQueueTimer.
BOOL CreateTimerQueueTimer(
PHANDLE phNewTimer,
HANDLE TimerQueue,
WAITORTIMERCALLBACK Callback,
PVOID ...
- 281
1
vote
1
answer
953
views
indirect return value address
I have usercall with calling convention I do not fully understand, it returns std::string but IDA recognize it as void.
I've noticed that every calle reads from x8/w8 afterwards, from wikipedia;
...
- 273
0
votes
1
answer
357
views
Why doesn't alloca_probe follow x64 calling convention?
I was practising reverse engineering on some Windows x64 applications when I came across this function:
call alloca_probe
This alloca_probe function has some strange implementation:
EAX is used as ...
- 3
1
vote
1
answer
166
views
Lack of arguments before a CALL instruction
I'm reversing an open-source windows application written in C++. I found in the disassembler the desired function that I'm trying to understand its behavior.
In the code this function is defined as ...
1
vote
2
answers
3k
views
IDA Hex-Rays: How to SAFELY fix incorrect function declarations?
Update:
The problem turned out to be more complex and complex because Hex-Rays incorrectly restores the stack after calls of stdcall functions from cdecl functions:
.text:00403F2F 074 ...
- 39
2
votes
0
answers
914
views
Show parameters stored in stack with __fastcall in IDA
This is an extension of my previous question. Here is a sample of assembly in x64 Windows executable using COM virtual function with __fastcall convention.
mov [rsp+32], rdx ...
- 1,559
2
votes
1
answer
333
views
What does __return_ptr do in IDA?
Let's say I have this program:
struct A {
int x;
};
struct A func1(int z) {
struct A result;
result.x = z+1;
return result;
}
int main() {
return func1(1).x;
}
When compiling ...
- 123
5
votes
1
answer
2k
views
How to specify stack-based return values in IDA Pro (GoLang)
tl;dr: I am reversing a GoLang 1.10 executable, compiled for Windows. I am trying to make IDA correctly recognize the calling convention.
Details: I am looking at an x86 executable where the ...
- 61
2
votes
2
answers
2k
views
Confused about the value of edx and eax registers used at function entry
I am currently using IDA to disassemble a keygen. In the first few lines of a specific function before the eax and edx registers are given a value, their contents are moved to stack variables var_40 ...
- 23
2
votes
2
answers
792
views
What compiler uses a calling convention that uses EAX as the first argument, then pushes onto the stack?
I recently asked to question: How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? After posting the question, I found that another function passed the first argument ...
1
vote
1
answer
174
views
How can an IA-32 program seemingly compiled with MSVC send its sole argument via EAX? [duplicate]
A C++ program I'm looking at appears to have been compiled with MSVC. It links to MSVCR100.dll, contains MSVC's security cookie code and uses the Microsoft mangling scheme. However, one of the ...
6
votes
1
answer
299
views
Returning value from faux DLL
A bit of background; I'm trying to inject my own code into the old game SimTower with the ultimate goal of reverse engineering it in the same way that OpenRCT2 was created.
Since SimTower is ...
- 165
8
votes
2
answers
1k
views
Register Calling Convention: written in stone, or in mud?
When disassembling an old Delphi 3 executable, I find some routines that pass arguments in registers EAX, EDX, and on the stack – but not in ECX!
For those routines, ECX never gets set to a '...
- 2,354
4
votes
1
answer
1k
views
How to find the stack and other info in a memory dump? ARM
I'm reversing a closed platform to try gain execution using an exploit, a stack overflow. I've been told that (since there are no debuggers) the best way is to use RAM dumps (that I have) to try ...
- 154
2
votes
0
answers
409
views
IDA Pro .til files backward compatibility
I read in the IDA Pro Book that
you may use till.exe to parse header files and create standalone .til
files which can be made globally available by copying them into
<IDADIR>/til.
This ...
- 1,550
2
votes
1
answer
804
views
How can I call a function that acts like __thiscall, except the caller cleans the stack?
I'm trying to call a function with a function pointer in C++, but I can't find any convention that fits it. Its stack arguments are (right to left):
func(float x, float y, int unk);
...but ecx needs ...
- 185
1
vote
2
answers
1k
views
Confused about calling convention in a Windows kernel mode driver
Reverse engineering a kernel mode driver (in its 32-bit x86 incarnation) I stumbled over what seems to be an odd calling convention. For a driver I'd expect to see __cdecl, __fastcall and __stdcall in ...
- 10.9k
1
vote
1
answer
768
views
Call a thiscall method from an injected DLL [closed]
I am trying to create a (32-bit) DLL to inject into a target process.
My DLL should call a function inside the original process. So far nothing special, however the target function is using the ...
- 111
2
votes
1
answer
214
views
Calling a function outside the module
Is it possible that a module will have a call to a function in a different module?
I know about import tables and delayed import tables. But are there any more options to do so? How will it look in ...
- 45
3
votes
1
answer
2k
views
Why is this _DllMainCRTStartup calling convention not as expected?
I've been looking around at various DLLs on my computer lately and I thought a decent place to start was the image entry point (typically _DllMainCRTStartup).
From what I can see of the CRT source ...
- 31
3
votes
1
answer
4k
views
How to create function prototype that recognizes arguments passed by reference in IDA Pro
I have a function prototype generated by IDA that looks like this:
However, it looks like there is a struct being referenced at the instruction lea edi, [esp+290h+var_240]. I would like IDA to ...
- 440
3
votes
2
answers
962
views
x86_64 Calling convention
I am slightly confused by some x86_64 linux code here (generated by llvm):
# BB#0: # %entry
push RBP
.Ltmp21:
.cfi_def_cfa_offset 16
.Ltmp22:
....
- 71
2
votes
2
answers
1k
views
How to programmatically identify binary calling convention?
I have a problem where I'm instrumenting stripped binaries; I don't know the start of main(). But there's always an init(), and init() calls libc_start_main(), which receives a pointer to main.
If I ...
- 163
2
votes
2
answers
2k
views
Change calling convention from IDAPython?
I have a 64-bit Mach-O sample that uses the System V AMD64 ABI calling convention (rdi, rsi, rdx, rcx,...); but IDA thinks all the functions are using cdecl. I'm new to IDA scripting but didn't find ...
- 21
15
votes
3
answers
11k
views
Which calling convention to use for EAX/EDX in IDA
I have some code (I assume Delphi) which uses only the EAX and EDX register for passing the arguments (and of course the stack if more are required). I looked which calling conventions would match, ...
- 963
9
votes
1
answer
4k
views
What x86 calling convention passes first parameter via ESI?
I am looking at some x86 code, which I believe was built using a Microsoft tool chain, and am trying to figure out the calling convention used during this call:
push esi ; save ESI (it gets ...
- 2,561
7
votes
3
answers
9k
views
IDA Pro function stack frame view
IDA Pro displays certain buffer or padding above (at lower addresses) local variables in stack frame view. For instance:
Example 1.
The following screen shot of stack frame view shows 12 bytes (...
- 3,088
8
votes
2
answers
2k
views
Identifying variable args function
How would a C variable argument function such as printf(char* format, ...) look like when disassembled?
Is it always identified by calling convention, or are there more ways to identify it?
- 4,785