5

I am reverse engineering a Windows game from 1995 using Ghidra, and I'm currently documenting all the decompiled functions. My goal is to replace a function in the original binary with a custom C function, test the game, and proceed incrementally, one function at a time, rather than attempting to get all the decompiled C code working at once.

Some challenges I'm facing:

  • The game was compiled with Watcom, so it's using its "non-standard" calling convention.
  • There are global variables everywhere, so the replaced function must be able to access those in the main executable.

I'm aware of projects like KeeperFX and Syndicate Wars port (http://swars.vexillium.org/), which managed to achieve similar goals (e.g., by converting the executable into a DLL).

What are my options for replacing a function in the original binary with a custom C function while addressing the challenges mentioned above?

3
  • @rce Yes, the "EXE to DLL" technic looks promising, but I don't know how they solve the shared global variables problem. Commented Apr 24, 2023 at 12:56
  • Watcom lives on as OpenWatcom ... check it out. If you want to get closer to the original, you could try 1.9 or 1.8, too.
    – 0xC0000022L
    Commented Apr 24, 2023 at 13:20
  • @rce: may not be relevant for this particular case, but there are some issues when using data exports and delay-loading together.
    – 0xC0000022L
    Commented Apr 24, 2023 at 13:22

1 Answer 1

0

When trying to renew an old game, it's important to have a working thing relatively soon - otherwise, there's a great chance of your work getting wasted. So it's good you've turned towards RE ideas with potential of having full game usable without having it fully remade.

The DLL approach taken in KeeperFX:

  • Works only if your executable has a relocation table - later versions of both Visual C and Watcom started to remove these. Without relocation table, the DLL will have ability to load only at one specific address in the virtual address space of the game - which means sometimes it will work, and sometimes not, depending on versions of other DLLs and their base addresses.
  • Requires you to do your C rewrite from main() and outer layers, towards the central functions which you will likely want to modify. Unless you will mix the DLL approach with some binary patching.

The ASM approach taken in SW Port:

  • Allows you to modify / rewrite to C any function at any time - no need to start with the main(). Because if you have the whole game disassembled, you can do calls back and forth between C and ASM.
  • Allows you to port to other OSes without a full rewrite - ie, SW Port supports Linux and Mac OS (as long as they run on i386 arch with 32-bit user space)
  • The 32-bit user space restriction will be easier to overcome in case OSes would stop supporting that. While it would require assembly changes, it is doable without a full rewrite.
  • Achieving the first runnable state this way is a bit more labour-intense that with the DLL approach.

There's also a 3rd approach - you'd find it in Magic Carpet HD. That project author just used Ida Pro to generate C code out of the whole game, and then continued fixing the generated C until it started working. I consider this approach very risky - you need to put a lot of effort to make the game work. Also, the generated C code has a very low quality this way, and it may negatively influence further progress of the project. Plus, the brute-force C conversion introduced a lot of bugs.

General points:

  • For both methods, you will find useful tools already developed in the respective projects (ie. export table updated for DLL approach, and C-to-watcom bi-directional calling wrapper for ASM approach).
  • The DLL approach required some specific conditions to work, while disassembling the whole game is a general approach which can be used for any game.

For accessing global variables and functions between original game and remade C code:

  • In KeeperFX, the variables and functions from within DLL were just exported with a prefix (_DK_ specifically), and could be easily used outside, like any DLLIMPORT. Calling convention change was handled outside of the DLL, as an ASM block within C. Not sure if there was a helper macro to hide the details.
  • In SW Port, any variables/functions which are used in assembly but also in C, need to be properly defined (as .global with decor) - the port source has macros for that already made. And for C-to-Watcom conversion, there is a tool which generates a wrapper C function - you need to add the function to the wrapper, and then you call it normally in C. The wrapper will get arguments from _cdecl call and put them into proper registers (you have to define the arguments when adding function to the wrapper).

Basically:

  • C variables are decorated, so to access ASM-defined vars in C you need to add that decoration to your ASM vars.
  • For C-to-Watcom calls, you get up to 4 vars from stack and place it into registers, and that's it.
  • For Watcom-to-C calls, you have to modify the assembly to just use stack and no registers.
  • While juggling the stack and registers, you need to mind clobbering/spoiling value within registers by functions. Watcom convention says - any register which was used as parameter, can have its value overwritten.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.