1

I'm trying to write to a certain file format directly out of an application. Although the file format itself is extremely simple, it contains a 32-bit checksum I don't know how to calculate.

I have access to an Win32 GUI application (quite old, coded in Delphi 7), that writes these files including the checksum. I already tried Ollydbg and IDA Pro Free to disassemble/debug, but I don't really know how to start.

Is there a way to trace the data? As I know the output file, I know the checksum and the input, is it possible to set a breakpoint "whenever the value 0x12345678 appears somewhere", and then I would be able to step back to find the algorithm?

Any ideas how I should start? Any good tutorial on this?

Edit: Thanks to the comments/answer I was able to do find a function that sets the checksum invalid value. I was also able to find two function that calls ReadFile. The first seems to read the header to see if the file seems to be ok, and the the other one is called by the function that sets checksum invalid in a loop, the file is read in small pieces and then processed. Unfortunately it does not call the function directly, the call stack shows, that from the functions that sets checksum invalid there are 6 functions called until ReadFile is done. Is there some way to trace the data, like a breakpoint everytime the buffer with the file contents is accessed?

Edit 2: I started to follow every command since the first buffer is filled, but it is horrible, I steped into about 100 functions for hours now and it has only ever accessed the first 6 bytes: Sometimes it reads data, calculates something, but that value doesn't seem to be used, single bytes are compared if the are LFs or CRs... Now I know why it takes a few seconds for that program to load a file of 10KB on an Core i7. It seems to read blocks of 128 bytes and process that data directly instead of checking the checksum first. Additionally there seems to be multithreading involved. I tried to set a memory breakpoint, but then the program crashes (OllyDbg has shown a warning that this could happen). Is it possible to do a backward trace "offline", I could set a start point when it first reads the file (I corrupted on purpose) and set a end point where it sets the checksum incorrect variable. It records all register/memory used in between and I could step back from the point where it sets checksum incorrect. Any idea?

  • Open the binary in IDA, xref the file write functions, and you'll find the place where it writes the data. From there, you go back, searching for where is that data created/modified, and you should find the checksum function. – Martin Oct 21 '16 at 22:20
  • You could try something like IDR it helps recognizing Delphi runtime functions. You can then use something like x64dbg, import the MAP and see what's going on dynamically. – mrexodia Oct 21 '16 at 22:22
  • In this kind of scenario, i often use procmon to check what the call stack looks like when the file gets written, which will give me an idea which functions are involved in writing the file, then give those a closer look. More information in my answer to this question. That is about an encrypted xml file, but finding the checksum algorithm would be quite similar. – Guntram Blohm Oct 22 '16 at 3:55
  • I feel this is becoming quite a different question and that existing questions might help you here if you search a bit. – NirIzr Oct 25 '16 at 4:20
5

Here are the ideas I came up with:

  1. Use CheatEngine's advanced memory analysis to "breakpoint whenever the value 0x12345678 appears somewhere".

  2. Use any decent debugger's (such as OllyDbg) to add a conditional breakpoint on NtWriteFile to break on condition having the specific value.

  3. Open the binary with a decent disassembler (such as IDA), find the imported file writing APIs and follow cross-references until you identify the CRC calculation code.

  4. Use signature scanning tools (such as findcrypt, SnD Reverser Tool) to identify any widely used CRC algorithms (you may be lucky).

If any of the methods is unclear I can elaborate.

Edit: reading it more thoroughly now, I see your edit could actually warrant a new question, but basically it seems the code you're dealing with is small enough to trace manually. That means keep hitting the "step into" and follow that code and values. You might want to do that several times if you're having a hard time follow the trace. Otherwise if you identify a buffer that's being written into multiple times you can memory breakpoint it. Another approach might be to open the file in IDA and manually inspect the code. IDA makes it easier to statically read code, and also allows has some debugging functionalities.

  • Thanks for your answer, I never reverse-engineered any software before (except finding strings in a hex editor etc.), so I'm still a bit clueless where to start. – Stefan Oct 22 '16 at 16:34
  • I'm sorry, this is stupid. A line-break does not seem to work in a comment, so now without any structure: 1. I tried CheatEngine, used the advanced memory analysis, but could not find out how to set a breakpoint there somewhere. I made a memory scan for that value, nothing found. 2. How do I do that? Is there a possibilty to scan for windows API calls? 3. "find the imported file writing APIs" - I have no idea how to that. 4. No success, I already generated lot's of checksums on the data. I assume the checksum is done only on parts of the data, size fields left blank... – Stefan Oct 22 '16 at 16:44
  • There is a string "checksum incorrect" that is printed when a file with invalid checksum is opened. I found where is it is printed, this function is called from what could be a switch, it is like mov eax, ds:dword_1234; sub eax, 2; jz loc_a; dec eax; jz loc_b; dec eax; jz loc_c; jmp loc_d. loc_c being the function that prints "checksum incorrect". That means that somewhere the value 4 is written to ds:dword_1234 in case the checksum is incorrect, how do I proceed to find that place? – Stefan Oct 22 '16 at 22:15
  • 1
    That is great info. I suggest you edit your question to include it :) – NirIzr Oct 23 '16 at 3:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.