Reverse engineer a checksum algorithm from application

Question

I'm trying to write to a certain file format directly out of an application. Although the file format itself is extremely simple, it contains a 32-bit checksum I don't know how to calculate.

I have access to an Win32 GUI application (quite old, coded in Delphi 7), that writes these files including the checksum. I already tried Ollydbg and IDA Pro Free to disassemble/debug, but I don't really know how to start.

Is there a way to trace the data? As I know the output file, I know the checksum and the input, is it possible to set a breakpoint "whenever the value 0x12345678 appears somewhere", and then I would be able to step back to find the algorithm?

Any ideas how I should start? Any good tutorial on this?

Edit: Thanks to the comments/answer I was able to do find a function that sets the checksum invalid value. I was also able to find two function that calls ReadFile. The first seems to read the header to see if the file seems to be ok, and the the other one is called by the function that sets checksum invalid in a loop, the file is read in small pieces and then processed. Unfortunately it does not call the function directly, the call stack shows, that from the functions that sets checksum invalid there are 6 functions called until ReadFile is done. Is there some way to trace the data, like a breakpoint everytime the buffer with the file contents is accessed?

Edit 2: I started to follow every command since the first buffer is filled, but it is horrible, I steped into about 100 functions for hours now and it has only ever accessed the first 6 bytes: Sometimes it reads data, calculates something, but that value doesn't seem to be used, single bytes are compared if the are LFs or CRs... Now I know why it takes a few seconds for that program to load a file of 10KB on an Core i7. It seems to read blocks of 128 bytes and process that data directly instead of checking the checksum first. Additionally there seems to be multithreading involved. I tried to set a memory breakpoint, but then the program crashes (OllyDbg has shown a warning that this could happen). Is it possible to do a backward trace "offline", I could set a start point when it first reads the file (I corrupted on purpose) and set a end point where it sets the checksum incorrect variable. It records all register/memory used in between and I could step back from the point where it sets checksum incorrect. Any idea?

Answer 1

Here are the ideas I came up with:

1. Use CheatEngine's advanced memory analysis to "breakpoint whenever the value 0x12345678 appears somewhere".

2. Use any decent debugger's (such as OllyDbg) to add a conditional breakpoint on NtWriteFile to break on condition having the specific value.

3. Open the binary with a decent disassembler (such as IDA), find the imported file writing APIs and follow cross-references until you identify the CRC calculation code.

4. Use signature scanning tools (such as findcrypt, SnD Reverser Tool) to identify any widely used CRC algorithms (you may be lucky).

If any of the methods is unclear I can elaborate.

Edit: reading it more thoroughly now, I see your edit could actually warrant a new question, but basically it seems the code you're dealing with is small enough to trace manually. That means keep hitting the "step into" and follow that code and values. You might want to do that several times if you're having a hard time follow the trace. Otherwise if you identify a buffer that's being written into multiple times you can memory breakpoint it. Another approach might be to open the file in IDA and manually inspect the code. IDA makes it easier to statically read code, and also allows has some debugging functionalities.