I have access to a few thousand binary files and I don't have the program that generated them. The files have been parsed and I also have access to the database where the parsed fields are stored. How can I reverse engineer the format of the files, given this information?
Short answer: You can't.
The files may be encrypted. If the encryption is strong, the implementors didn't make any stupid mistakes, and the key is hidden in the original binary, then there's no way to match your files to the readable content.
Longer answer: It might not be that bad.
First, try to find out some general information about the files.
Do they all have the same size, or are the sizes different? If the sizes are different, might that be explained by the file contents, or would you expect all sizes to be equal?
Is it possible the files are compressed? Try making a .zip of them. If zip is unable to compress them, they probably already are; in this case, try to find out which algorithm has been used on them. Binwalk might be helpful for that (the web site says firmware analysis, but it will recognize many compressed file formats just fine). If they are compressed, and you can't find out the algorithm/decompress them, you're stuck.
Open some of the files in a hex editor, and/or run the strings utility over them. Do the files have parts that look readable?
Try matching file names, or file contents you identified, to database rows. Knowing what is in a file helps a lot in identifying specific content.
If all of the files have the same size, or almost the same size, try finding out which parts are equal between them and which are different. I did something similar on different blocks of the same file in my answer here, this might get you started.
If there are some numbers that you think should be in a file, convert them to integer/long/float/double representation, and search for these bytes in a hex editor. Repeat with a few different files. With a bit of luck, you'll find the same information at the same file offset, or in a part of the file that's identifyable, in a few cases.
If you can (copyright issues etc.) tell us the name of the software, and upload some example files together with their decoded content. A forum member might know the software, or recognize the file format. And some of use just like this kind of challenge as well.
First of all, the question is too broad. Examples would help a lot.
Here is an algorithm I'd start with without specific knowledge about files.
- First of all I'd try to answer to the question is it standard format. Using linux file utility on the files may help. If this file appears to be standard I'd try to find documentation on the format and/or to reverse engineer a tool that works with it.
- If the format is not standard I'd try to check the structure of the file. Does all the binaries have similar header ? If yes, where are they differ ? Is it related to file size, number if entities in it, creation date, modification date, etc ?
- After that I'd start to dissect this files to elementary types. Searching for strings related to the data may help to understand a lot. Searching for known values (rarely occurring integer constants for example) may help to connect between these constants and their locations in the file.
- After that I'd try to find almost similar files. Where are the differences ? How this differences related to the data ? If you have a file with a small amount of known differences, you can make binary diff and see where exactly these differences stored.
After performing all these actions you'll probably enough information to ask more specific questions.