If the surrounding window seems to be a genuine window, but the individual buttons are not, I'd assume that
- The surrounding window displays a bitmap, or does whatever else is needed to display the individual buttons
- The surrounding window reacts on mouse clicks
- And, the surrounding window determines itself which "button" is used by looking at the mouse coordinates at the time of the mouse click.
So, what we want to do is, start the program in a debugger, and use a breakpoint that is triggered when the mouse click occurs. The problem is, the window procedure gets called a lot of times, with all kind of messages, while you want to break on some of them (for example, WM_LBUTTONDOWN), but not on most of the others (for example, WM_PAINT).
- Ollydbg can do this directly - place a conditional breakpoint that triggers depending on the window message.
- If, for whatever reason, you can't use that feature, you need to disassemble (and partially understand) the window procedure. It will receive 4 parameters on the stack, the 2nd of which is the window message (see Using Window Procedures). The window procedure will, at some point, compare this parameter to the values of well-known window message IDs. You might be interested in
WM_LBUTTONDOWN (0x0201) or one of the other Mouse Input Events). Check where the code branches to when it receives one of those "interesting" messages, and place a breakpoint on that branch.
After placing a suitable breakpoint, run your application in the debugger, click the mouse, and hope the debugger actually hits the breakpoint. If it doesn't, check which mistake you've made.
Single-Step your code. It will probably fetch parameter 4 from the stack, which encodes the X and Y coordinates of the mouse, and compare these parameters to something - chances are, there's an array of structures that defines rectangles and jump addresses, and a loop that checks against each entry.
Once you've found that, you're finished.
Sounds too easy? It probably is, because, depending on your framework, the actual comparisons might be buried deeply in stuff that was a bunch of C++ classes in the original source, and there might be various calls, some of them indirect (in the case of C++ class methods) until you get to the "interesting" position.
Sometimes it's easier to use Ollydbg to resolve this. Single-Step the source after your breakpoint, using "step over" to step over function calls. Whenever you see something interested happened within the function call, note the address. Restart your program, run to your breakpoint, run to the address you noted, and "step into" the function this time. This will allow you to put the finger on the interesting code relatively quickly, while stepping over all the strcmp() and malloc() and WriteALotOfStuffToSomeFileIfDebuggingIsOn() functions.
Sometimes it's easier to use IDA. Generate a call tree from your window procedure, focus on the first 3-5 levels. Search your code for occurrences of magic numbers, like the 0x0201 above. Check if one of these is a CMP XX, 0x0201 in one of the functions in your call tree. This might be just the position you're looking for.
Make a screen shot, and paste it into some image viewer program (I like Irfanview for that). Get the pixel coordinates of the button within the frame window. Search, in the data section, for those values, surrounded by what looks like a table of similar data. This might be your coordinate/jump table. If you find something, place a hardware breakpoint on it to find out when that memory gets accessed, or use the IDA cross reference list to get respective addresses in your code.
stringson it. Study the application behaviour.