2

for hobby I'm just trying to reverse and old DOS game executable.

I'm quite new with reversing so I'm having hard time understanding some bits about IDA Pro. I let the program choose the best-fit strategy to disassemble the executable and it seems to work fine but some procedures are left unrecognized (marked in red) as in the picture:

enter image description here

Now I see that I can explicitly mark this as a function but then I'm not able to find any xref to it (also because I guess IDA would already have turned them into procedures if it was the case).

So I was wondering what is the exact purpose of these functions, are they called by dynamically generating the address to them or what? Like a CALL instruction to the content of a register.

A side question: I see that the binary had been split into 4 segments category:

  • code (clear intent)
  • data (clear intent)
  • stub
  • overlay

While the first two, coming from a C/C++ environment are quite clear I'm trying to understand exactly what the other two are.

If I understood it correctly stub segments are just bridges to procedures which are far from the current executing segment, while overlay segments are segments which are replaced at runtime over the same piece of memory to let the program work with much more code than the addressable space. Is there any good tutorial about this kind of memory management in the old times?

2 Answers 2

5

Unused functions in executables

There are various explanations for that.

Obfuscation, as Itsbriany said, but i don't think that was much of a thing in old DOS executables.

Unneeded library functions. If your code links in, for example, the math library (those old processors typically didn't have math coprocessors for floating point), you'll typically get all of the math emulation. Your program might use sqrt for distance calculations, but never use sin, cos and similar trigonometrics, but they get linked in anway.

Function pointers. If your original C program had something like int (*funcs())[]={func1, func2, func3}; int result=(func[index])(); and these are the only references to func1, func2 and func3, then ida won't recognize them as being used. However, you'd find a data xref in them.

C++ object methods, although when C++ started getting used, 32 bit processors were already common and you'd compile your program to a 32 bit binary using a dos extender. If methods never get called directly, only through the vtable of a class, then the same thing applies as with function pointers.

Debugging functions in the original source code that just never get called in the finished binary. If the programmer did something like

#ifdef DEBUG
some_function(parm1, parm2, another_parm);
#endif

and forgot to put some_function into #ifdef as well, then the finished binary will still have some_function, but no call to it.

More than one programmer working on independent parts of the code. One programmer might have been assigned to the task of reading/writing a config file, and he implemented read_config() and write_config(). Later, someone decided that the game should only use read_config(), and the configuration part should be done in a separate setup program. If both functions are in the same source file, they'll be compiled into - and linked in from - one single object file anyway.

Overlays in old dos programs

I don't know about a good tutorial, but here's basic info about overlays:

You needed to identify parts of your code that didn't require each other when running. For example, your program might have a bunch of functions to navigate a world map, a different bunch of functions to navigate a dungeon, and a third bunch of functions to interact with a shopkeeper.

You'd compile those separately, so you get 4 files: main_game.exe, world.ovl, dungeon.ovl, and shopkeeper.ovl. (Oops, there was an 8 character limit to filenames.) main_game.exe would be loaded into low ram, and calculate a segment for the overlays to load in - all at the same address. The easiest way to do this is make a segment that has only one variable, which gets loaded last - the address of that variable would be the address to load the overlays to. Let's name it overlay_start.

To connect the main .exe to the overlays, each of the overlays would have a jump table at the start. For example, in world.ovl:

0000 jmp show_world_map
0004 jmp draw_player_on_world_map
0008 jmp draw_enemies_on_world_map
....

and in dungeon.ovl:

0000 jmp show_dungeon_map
....

Now you needed a bunch of functions to link into your main program to call these overlay functions, which all looked basically the same:

extern int (*overlay_start())[];

void show_world_map() { load_overlay("world.ovl"); overlay_start[0](); }
void draw_player_on_world_map() { load_overlay("world.ovl"); overlay_start[1](); }    
void draw_enemies_on_world_map() { load_overlay("world.ovl"); overlay_start[2](); }

void show_dungeon_map() { load_overlay("dungeon.ovl"); overlay_start[0](); }
....

void load_overlay(char *filename) {
    /* omitting all the error checking, loops for more than 64KB, etc.) */
    FILE *fp=fopen(filename, "rb");
    fread(overlay_start, 1, SOME_LARGE_NUMBER, fp);
    fclose(fp);
}

(I omitted the function parameters in this example to keep it concise).

Depending on your compiler, much of that could be auto-generated. And these functions, possibly including the load_overlay function, is typically what goes into the stub section. It's just a bunch of functions that keep the linker happy when linking main_game.exe, make sure the correct overlay is loaded, call the intended function, then return.

2
  • Thanks, I think I got it clear, actually I think the whole game just runs in real mode and pages are swapped accordingly by the OS (DOS). The game is said to require 2.7Mb of EMS, I studied a little the mechanics behind this. Is it possible that everything is just in real mode (so no extenders) and overlays are just swapped from EMS to conventional memory on requests made from the executable to DOS?
    – Jack
    Nov 30, 2015 at 16:13
  • DOS doesn't handle EMS; instead, your EMS device driver does that. Which means you can't have DOS load the overlay into EMS directly, you still have to do it yourself. But of course, that's just a load_overlay function that's a bit more sophisticated. a) when an overlay gets loaded that's already in EMS, load it from there; b) when an overlay gets loaded that isn't in EMS, load it from disk; c) when an overlay is "unloaded", copy it to EMS first. This avoids some gory stuff as well like static variables being reset to 0 every time the overlay gets reloaded, and it provides a great speedup. Nov 30, 2015 at 16:38
1

The code you are looking at is likely packed because it has that stub segment.

Software vendors and malware authors pack their executables to make reverse engineering more difficult since most of the binary will appear as "compressed" data (non-machine instructions). Only the very first instructions should be apparent and will likely not be in the .text segment.

If that is the case, then the binary will unpack itself during execution. This unpacking procedure will "decompress" the "compressed" data and reveal more valid machine instructions which will later be executed. The unpacking stub could also set global variables or dynamically allocate memory that will potentially hold function pointers to that procedure you are looking at.

Additionally, the code in the .stub segment should contain many mathematical instructions such as xor reg1, reg2 (where reg1 and reg2 differ), shr, shl, ror, rol, etc... with many jumps.

Again, you are not seeing any xrefs to that function because IDA cannot find that subroutine called in the disassembly because it is possibly referenced by a global variable or via a dynamic allocation on the heap during the unpacking procedure.

There is also a small possibility that the authors of that executable forgot to call that procedure.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.