Extracting the contents of an unknown archive file format

Question

I need some help and advice here. I am privately learning on how to reverse engineer our teachers custom file archive file that have other files stored inside of them. for school project.

But, I am kinda lost on where I need to start with the file. What do I look for in the file after loading it into ida-pro To, be able to create my own self extracting Tool for my project.

The File format is .crackme that has been compressed with files inside of it from our teacher at school but when I double click on the file it says the file is "archive either unknown format or damaged".

So, yeah the file extension he gave us is not any known on the web he created his own extension for us to crack it and make an extractor tool to get all the data from the archive his extension name is ".crackme" so yeah.

Any Advice, here any help to is much appreciated by me.

Answer 1

I hope you'll find your experience in RE.SE enjoyable and educating :) Reverse engineering is often studied as a hobby, so you're in good company!

IDA pro is a disassembler, which is focused on reverse engineering / reading assembly/machine code. that is - the code actually executed by the CPU. Since it appears you were only given a format file, without any executable code to manipulate / process the file, IDA will prove less useful.

Instead, the only tool you should require for such a task, assuming the format is indeed proprietary, is a hex-editor / reader. 010 Editor is a great one, and it has a trial version.

To eliminate the possibility of a known format, try running a couple file identification tools such as the linux file command, TrID, binwalk, etc. These will try to identify common file formats (binwalk is specifically focused on archive formats), which if provide any insights should point you in the right direction.

To reverse engineer a file format solely based on a single file content, you'll need to map out the the file's apparent structure and you don't have a lot to go with. Excluding textual and very simply file formats, this should be kinda difficult without any additional resources (more files, a program that handles the format in any way, etc).

Here's a list of pointers/tips, although you'll probably find more information available online:

1. As most files begin with a header, you should begin with mapping what are the different members of the header structure.
2. Things that are magic values, strings, offsets and sizes are quite easy to recognize. Most decent hex editors will show different representations of every few bytes, so it'll be easy to recognize.
3. Start by dividing the bytes to different members, before you try to understand their meaning. For example, given the following hex stream: 00 05 00 00 00 01 BE F1 CA D7 it is easy to notice the members are a word (2 bytes), a dword (4 bytes) and another dword. It is also likely that the first two are integers while the third is either a magic value or a CRC.
4. If the file has any recurring delimiter values, those should also be easily identifiable (although less likely to be present in an active file format).
5. If you manage to recognize any other file headers (probably using their own magic values), those are good leads to figuring out where the actual file content is placed. This will also mean the archive file does not compress the contained files, linux's tar, for example, can do that. Alternatively, if you encounter long streams of smooth or random looking binary streams those can be the actual compressed data. I tend to believe your teacher did not invent he's own compression algorithm, so there you'll need to find similarities to other compression algorithms, perhaps running those streams under TrID and file again could help.
6. Common looking patterns, for example, may be a useful way to uncover recurring structures (for which you'll have more than one reference point, yay!). For example, if you recognize every file-name string is followed by a sequence of bytes with a similar structure, those are probably two instances of the same structure.

Answer 2

Many archive formats contain headers. gzip headers for instance begin with 1f 8b. Use a hex-editor to check for known magic numbers in the header first.