I am a developer for the Hachoir project, which aims to describe the format of any file down to the bit-level. Currently it has parsers for a number of well-known formats, including Java classes, Windows executables, MP3 files, etc. The full list of parsers can be found in the subdirectories here.
When working on a new parser for hachoir_parser
, it is often necessary to find information about a file format. There's no single source describing every format (even as Hachoir aspires to be this source, it is not nearly comprehensive enough). Generally, the first step is just to search for <file format name> file format
, e.g. java class file format
, and look for documentation on official sites (for java classes, this turns up Oracle's documentation, which should be all you need). If there are no official sites, you may still turn up some documentation from someone who has worked on the format in the past.
For common file formats, this turns up the format specification you want about 90-95% of the time. Larger software companies, like Oracle and Microsoft, post their file format specifications online for interoperability purposes. For example, you can find documentation for PE (Windows EXE/DLL), MS Office formats (XLS, PPT, DOC), and other Microsoft formats by browsing or searching MSDN.
For multimedia formats, the Multimedia Wiki is a great resource. They also cover some game file formats as well.
For a less common file format, for which I do not find a specification (or suitable description) from Googling, my approach is usually to find an open-source program that does understand the file, and either locate their format specification source (if described in a commit or README), or read their source code directly to understand the file format.
If there are no open-source programs for the file, and no openly available descriptions of the file format online, the file format is probably quite obscure. For game files (in which many developers insist on using their own proprietary formats), I've found XeNTaX to offer some good pointers and a good community to help figure out the formats. With other kinds of formats, you may have to start examining the samples you have to compare the byte fields and elucidate their function. If you have a program that accepts these files, you can try changing the fields methodically to determine what effects they have on the program's output. This is ultimately the "real" reverse-engineering work, and I think it is not within the scope of this answer.