Now I most definitely could be wrong considering I am pretty new to hacking files like these anyways, but In hex editors, I always open files to what I think is an encryption. Now I don't know if these files are actually encrypted or maybe they are in a programming language I'm not aware of, but is it possible to "decrypt" said files?
Definitely not just use the hex editor. You should use the debugger OllyDbg or ImmunityDebugger, and bring the following questions:
- What language it was written the program
- When the program performs the decryption of the file
- What happens before and after the file has been decrypted
- Which API could use
- Both with OllyDbg that with Immunity can see all the intermodular calls
- At this point you can set breakpoints and debug APIs found
The process is not immediate, you must first locate the piece of code that performs the decryption of the file to make a good analysis.
I depends on the format the data is in
Java Files: are compiled to java bytecode and run in a vm. Java bytecode is not easy to read or understand but there are programs which can decompile java bytecode into java source files.
Binaries: These are compiled programs. It is impossible to automatically convert this back to source code but you disassemble it.
JSON/YAML: These are commonly used to store structured data. You can view them in a text editor.
If you want to reverse engineer a game first learn to program. Opening files with hex editors won't get you very far unless you understand what is happening.