I'd like to manually "pull out" information of files of a specific file format. Needless to say, they were encoded in some form or another - I couldn't access a "plaintext" version of them. However, the program associated with the file obviously is able to read them - And I was wondering if I could somehow reverse engineer that program and thus find out how they're being encoded, to read the content myself.
It's a broad question - But I'm not sure where else to start.
This is what I would do given the very limited information:
I would take the above information gathered and search the Internet for any constants or patterns that I found across the files. See if you can find anyone else discussing these files or any programs which use them. If you can find other programs using the files, try and locate source code or something on GitHub.
As you said, this is open-ended and there isn't much room for discussion here other than comments, but this is how I would start the process. Do you know which program is encoding the files? If so, get ahold of the program's executable and use a tool like PEiD KANAL and/or signsrch on it to search for common encoding algorithms, then get the addresses of those algorithms and open the program executable in a disassembler and examine the memory addresses. Last but not least, another trick you can use is if you can figure out which type of info may be in the file (such as URLs), you can use the Didier Stevens tool called XORSearch and search for "http" and the program will try a bunch of different XOR keys for that pattern and if it is able to crack an http somewhere, you may be able to find out the encoding that way.
I would advise you to use a dynamic analysis of the software by tracing it using tools such as the Intel PIN tool (I am assuming you are dealing with x86 assembly).
With this tool you can trace each executed instruction by the software (see this tutorial on PIN).
Then, your point will be to find system call that open the correct file (one of the parameter of the system call will be the path to the file, so you should be able to find it with a simple grep).
Once you have located the instruction that open the file, I would advise you to start disassembling from this point (use software such as IDA pro, Radare2, or even gdb/Ollydbg could do it).
Try to take a look at strace and syscall monitor program likes .
With those tools it will give you more information and details.
China has a folk adage："If a worker wants to do a good job, he must sharpen his tools first".
