I hope you'll find your experience in RE.SE enjoyable and educating :)
Reverse engineering is often studied as a hobby, so you're in good company!
IDA pro is a disassembler, which is focused on reverse engineering / reading assembly/machine code. that is - the code actually executed by the CPU. Since it appears you were only given a format file, without any executable code to manipulate / process the file, IDA will prove less useful.
Instead, the only tool you should require for such a task, assuming the format is indeed proprietary, is a hex-editor / reader. 010 Editor is a great one, and it has a trial version.
To eliminate the possibility of a known format, try running a couple file identification tools such as the linux
file command, TrID, binwalk, etc. These will try to identify common file formats (binwalk is specifically focused on archive formats), which if provide any insights should point you in the right direction.
To reverse engineer a file format solely based on a single file content, you'll need to map out the the file's apparent structure and you don't have a lot to go with. Excluding textual and very simply file formats, this should be kinda difficult without any additional resources (more files, a program that handles the format in any way, etc).
Here's a list of pointers/tips, although you'll probably find more information available online:
- As most files begin with a header, you should begin with mapping what are the different members of the header structure.
- Things that are magic values, strings, offsets and sizes are quite easy to recognize. Most decent hex editors will show different representations of every few bytes, so it'll be easy to recognize.
- Start by dividing the bytes to different members, before you try to understand their meaning. For example, given the following hex stream:
00 05 00 00 00 01 BE F1 CA D7 it is easy to notice the members are a word (2 bytes), a dword (4 bytes) and another dword. It is also likely that the first two are integers while the third is either a magic value or a CRC.
- If the file has any recurring delimiter values, those should also be easily identifiable (although less likely to be present in an active file format).
- If you manage to recognize any other file headers (probably using their own magic values), those are good leads to figuring out where the actual file content is placed. This will also mean the archive file does not compress the contained files, linux's tar, for example, can do that. Alternatively, if you encounter long streams of smooth or random looking binary streams those can be the actual compressed data. I tend to believe your teacher did not invent he's own compression algorithm, so there you'll need to find similarities to other compression algorithms, perhaps running those streams under TrID and file again could help.
- Common looking patterns, for example, may be a useful way to uncover recurring structures (for which you'll have more than one reference point, yay!). For example, if you recognize every file-name string is followed by a sequence of bytes with a similar structure, those are probably two instances of the same structure.