If I'm trying to figure out how a program accomplishes a certain task or how it works with an unknown data structure or format, what techniques and tools can I employ to quickly and efficiently locate the binary sections of interest to start answering those questions?
As a concrete example, say I'm trying to figure out how a game client calculates and validates a checksum. I've observed the following properties without any initial disassembly:
- That checksum is used to validate whether an ingame item stats is legit or not.
- The checksum is being performed on the client-side.
- It is a 2-byte checksum.
What I've done is to feed in a particular item with my own checksum signature so it could be identified in memory when I disassemble it. Once I got the location in memory I set a memory breakpoint on it whenever it gets accessed. From there I just backtraced where the control flow was coming from.
While it took a long while with a lot of trial-and-error I was eventually able to locate the code section responsible for this checksum calculation and successfully recreated a C-program that does this.
Another problem with the approach I used is that there could be a lot of 'false positives'. That memory location I'm monitoring could be accessed by various parts of the program that has nothing to do with the checksum calculation -- thus where the trial-and-error comes in.
The focus of asking this question is, is there an easier way to go about this? What if I wanted to identify key data structures being used by the program? What about identifying functions and calculations that aren't as 'easily observable'? Not all values being used in the program can be easily isolated using a memory scanner like cheat-engine.
