With a hex editor, can you really examine/modify every piece of code/data encoding in a "file"?

Question

Say I open a 1x1 BMP file with a hex editor. Will the values therein link directly to the specification used to encode the image, and what the decoding image program will also see?

If a dig in to Sonic The Hedgehog's bin file, can I really change every single entire encoded instruction using Motorola 68000's binary opcode map specification (assuming the assembler used will place the exact binary opcodes for the correct instructions taken by a Motorola 68000), and alter data values, change instruction positions, etc.? Basically, at the byte-level, I am exposed to the entire mass of a file structure, and have the control to change it at any rate I want. Basically, you can think of this question as this, to be blunt:

"Is there anything that CAN'T be accessed from a hex editor in a file, or something hidden?"

Is that correct?

Answer 1

You can open any files in a hex editor. The way to think about this is that programs that read files are developed specifically to interpret the bytes in a certain way. The header in a file might contain specific data that only a specific program can understand and use.

There are also some files that are encrypted, and a program with a certain algorithm can decrypt it. Like a game ROM, an emulator is just a program that knows how to interpret its bytes and render the information in the form of a game.

If you were to manually modify the bytes in a file, you can corrupt the intended information needed by the program. However, nothing can stop you from manually creating your mp3, game, document from using only a hex editor. It would only be painfully hard.

Answer 2

You can edit anything you want with a hex editor. There are ways to "hide" entire files but as soon as you can make them visible and open with a hex editor, then there are no ways to make parts of a file invisible. (1)

However ... You need to work within the frame of what you are changing. Suppose you edit a simple BMP image, and you "change" its size from 1x1 to 2x2. Will that "work"? No -- you are changing an important property (the image size) but not the actual image, and no program can read your modified BMP image anymore, since it expects a 2x2 image now.

However, if you only change the color of the one pixel inside the image, then there is no problem. BMP files do not contain a checksum, and so your altered file will be read correctly.

The same goes for your "Sonic" example. Yes, you can modify single instructions -- but (within the constraints of the entire executable) you cannot 'insert' or 'delete' them. And before you can change a single instruction, you need to be absolutely sure it is a single instruction, and not part of a previous multi-byte one. This might be hard to see if you are looking at a raw hex dump.

A third problem may be that the file is encrypted or otherwise protected against tampering (e.g., with aforementioned checksum). If that is the case, you can look all you want, but you cannot find anything useful to change; or, changing the data may make the program stop running (i.e., if it detects the original has been modified).

(1) Back in the days of floppy disks and CD-ROMs there were ways to invalidate disk sectors. That way, a 'file' could not be copied to another medium or opened with regular disk editors. Of course, the reading application "knew" which sectors were damaged and did not try to access them.

Answer 3

In general you can access any byte written on disk or in a file however there can be places you don't see if you don't look for them.

1. There are for example resource forks (Apple, HFS) or alternate streams (Windows, NTFS) that are not displayed by most hex editors when you open a file that has such extra information. See also Wikipedia - Forks
2. Some information can be hidden on areas of a file system that are outside file boundaries. This can work because most file systems store files in blocks and the last block of the file can have some extra bytes not accessible to a hex editor. For this case a disk editor helps.
3. There's the whole world of steganography - extra information can be encoded in such a way into other data like images that you don't see a difference (visually). Of course you can change any byte in such a file but don't notice the "hidden information layer".