What ways is there for an application to detect that it has been modified?

Question

I am using ollydbg and a Hex editor. I confirmed that once the application is edited in any way it behaves different than normal.

My first thought was that the file is checking the checksum value so I looked at the intermodular calls in olly and did not see anything about checksum. I was specifically looking for MapFileAndCheckSum

But I am trying to reason this out, I am thinking that a checksum value has to be hard coded in the file so it can be compared to the actual checksum. So I am wondering from the developers point of view how is it possible to get the checksum value to be hard code when the application is not complete/compiled

Which brings me to the question. What ways is there for an application to detect that it has been modified?

------- EDIT ------- Additional Information ------

I have been doing some testing and I have to say I'm baffled as to where the checksum value is being stored.

1. There are no connections to the internet.
2. Only one dll comes with the application (I extracted the installer files manually) the dll file is old and was last modified prior to the application. I even compared it to an earlier version of the application that did not have this checksum check and the dll is identical.
3. I taught that maybe the checksum value would be entered in the registry by the installer so I extracted the .exe and .dll to a separate computer that has never used the installer. Changes are still being detected!
4. It is definitely a checksum test, as I have changed a single byte of padding by from 00 to 20 and the change is detected. If I edit back to 00 to application performs normally.

So now I am wondering would it be possible to calculate what the checksum is going to be before entering the hard coded checksum value? I do realize that the actual checksum value will change when changing the hard coded checksum value. I want to know if there is any method to predetermine a checksum value when hard coding and finding a match. Seems impossible but I cannot think of any other means considering the situation.

Answer 1

The application can check only a part of itself, excluding the checksum part. You can also have a runtime decryption, that will be wrongly decrypted if tampered with. There are several research projects in this area.

Answer 2

I'm assuming you've hex edited the actual file. In that case, the application may be reading itself from disk and manually calculating a checksum. Most likely it uses one of the following to load the file into memory:

• CreateFile/OpenFile with ReadFile (and all their suffixed forms)
• CreateFileMapping and MapViewOfFile (and all their suffixed forms)

You can use Process Monitor to conveniently monitor file-related API calls and filter by process name, or just bp everything.

Also remember that a checksum could be stored in dll's/any other files that come with the application, or downloaded from the internet.

Answer 3

MapFileChecksum is used to calculate the checksum of the executable that's stored in the IMAGE_OPTIONAL_HEADER structure using Microsoft's custom checksum. It's usually used to re-calculate the checksum value once the executable has been modified, since the windows kernel checks the checksum before it loads drivers and system files. But in case of your application, the application might be using any kind of checksum (crc32, adler, e.t.c.) or might even be using hash functions. So just breaking on MapFileChecksum might not be enough. Since it has to read it's own binary to calculate the checksum/hash, break on the file manipulation functions and go from there.