0

I am reversing an old DOS game from 1989. What are the function calling conventions for the 16-bit 8086 processor that is used to run this type of program?

How it works on Windows x64

By function calling conventions, I mean, what registers are used to pass what function parameters? Example:

int test(int a, int b, int c, int d, int e, int f, int g) {}

In Windows x64, this would correspond to

rcx, rdx, r8, r9, [stack], [stack], [stack]

How it works on 8086

I took a quick look at the X86 calling conventions Wikipedia article, and 8086 looks more complicated than x64. It looks like there are different kinds of calls cdecl, fastcall, etc., and each type of call has its own conventions.

8086 calling conventions table

A quick peek in Ghidra at an old MZ executable confirms that multiple types occur in the same binary.

Ghidra function calling conventions

Questions

  • Can somebody provide a summary of the different types of 8086 calling conventions, their use cases, and the registers they use?
  • Which type is most common?
  • cdecl has an empty register column in the above table... does that mean it passes everything on the stack?
  • Do these "right to left" conventions push everything onto the stack in reverse? g to a instead of a to g?
  • What is Ghidra stdcall, regcall, thiscall? Does one of these correspond to fastcall in the Wikipedia table?
2

Calling conventions are compiler specific, meaning they are specified by the compiler manufacturer like Microsoft, and not by the 8086 manufacturer like Intel. In case you program in Assembler or build your own compiler, you can apply your own method to pass function parameters.

Nevertheless, in the C/C++ world, there are mainly the __cdecl and the __stdcall conventions.

cdecl: Parameters on the stack, rightmost parameter first (RTL). cdecl calls need slightly more code than stdcall, because the caller has to cleanup the stack after the call. Thus, when reversing you might make a guess about the programming language from how the stack is cleaned after a function call.

Pascal: Convention originally used by the Pascal programming language, parameters on the stack, leftmost parameter first. The Win32 API calls (it is a C interface) were all tagged with the PASCAL keyword.

stdcall: same as Pascal

Both (cdecl and stdcall) are the widely used, as everybody uses C Compilers and the Win32 API is also very common (although not everybody uses it...). I do not know if other compilers use the same keywords. It might be noted that using a DLL in your software you must know the calling convention of its API, as it must match the one of the caller. This is a frequent cause of trouble when using DLLs you didn't create yourself.

The Ghidra variants 16near and 16far of these two are to indicate the segmented memory model (in contrast to a flat memory model) of the 8086 architecture. 16near indicates intra segment (within 64k), and 16far indicates inter segment (call into another 64k segment). These have their origin in the 8086 architecture, where different assembler calls for near and far calls are present.

thiscall indicates a call to a member function (method) of a C++ class in the Microsoft compiler. In the 32-bit world, the ECX register holds the "this" pointer in that case. Don't know if this existed already in 16-bit world. MS released their first C++ compiler in 1992, thus you will not be bothered with your game with __thiscall (if built with a MS compiler).

Not sure about the exact meaning of regcall and fastcall, but I also assume a parameter transfer in registers.

Please read all this with a grain of salt, the 16-Bit times are (at least on PCs) long gone. And if you don't bother to google about these items, you surely will find a wealth of information.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.