This is what I would do given the very limited information:
- Start by examining the file in a hex editor like HxD or 010 Editor. Take note of any "constants." For example, all Windows exe files start with the bytes 4D 5A or "MZ". Do these files start with a consistent byte signature? Scroll down in the hex editor and find out what other patterns you can find.
- Use a tool such as Detect It Easy or PortEx Analyzer to measure the entropy on the file's bytes and/or map them out. Is the entropy very high such that the tool indicates the file is packed, or is it lower? This can give you clues on whether or not is encoded and perhaps even what type of encoding may be used.
I would take the above information gathered and search the Internet for any constants or patterns that I found across the files. See if you can find anyone else discussing these files or any programs which use them. If you can find other programs using the files, try and locate source code or something on GitHub.
As you said, this is open-ended and there isn't much room for discussion here other than comments, but this is how I would start the process. Do you know which program is encoding the files? If so, get ahold of the program's executable and use a tool like PEiD KANAL and/or signsrch on it to search for common encoding algorithms, then get the addresses of those algorithms and open the program executable in a disassembler and examine the memory addresses. Last but not least, another trick you can use is if you can figure out which type of info may be in the file (such as URLs), you can use the Didier Stevens tool called XORSearch and search for "http" and the program will try a bunch of different XOR keys for that pattern and if it is able to crack an http somewhere, you may be able to find out the encoding that way.