Identifying the nature and contents of a file

Question

As a preface, I do not have any experience in reverse engineering. I am an application developer, so this field is way out of my area of expertise. My end goal is to create a web page out of this information but this is the first hurdle I faced: getting the required data in the first place.

I am currently trying to identify the nature of a file that I have suspicions might be encrypted and/or compressed (I'm still trying to learn the tell-tales of a file being encrypted/compressed). Since this is way above what I can do at the moment, it would be great if someone can help out or at least provide some guidance.

This is an example file (do let me know if you'd like some more examples):

https://drive.google.com/file/d/1vwt4G55E-TaGvuWM86L6S-7Sbd2o0gOp/view?usp=sharing

So far, here's what I know and attempted:

• File is most likely a JSON or CSV (not 100% sure, but all the other unencrypted metadata files are JSON/CSV format)
• File has a lot of repeated sets of binary ef bf bd, which I'm assuming are just paddings or an attempt to obfuscate?
• Most of the corresponding hexes do not really translate to a readable set of standard encodings such as UTF-8

Any help/guidance would be appreciated.

Answer 1

When given an unknown file there are several things you can do:

1. Identify if the file is a binary format (raw bytes) or a text format. Any hex/text editor should be enough for that purpose.

2. If the file is Binary - it could have text strings inside, which could help understand the nature of the file. In your case, no text strings appear in the file.

3. Some binary files have "magic values" - identifiers that help to understand which file format it is. for example - the 4 first bytes in ELF file format will be always .ELF. In your case, the file starts with CE A3 EF BF which doesn't correspond to any known format.

4. Next step will be to use binary tools like binwalk to identify any compressed/internal files/objects. In this case, it yields nothing.

5. Next step will be to calculate the entropy of the file. It could be done by binwalk -E <file>. in this case, it yields 0.52 - the higher the entropy it, the higher chances the file is encrypted/obfuscated. Here, it looks like the entropy is not high, meaning the file is not encrypted/obfuscated.

So overall it looks like it's just a custom binary file format. In order to continue investigating, you need to understand which program is using the file - and in what purpose, and continue with a reverse engineering process, usually based on static disassembler like IDA PRO or the free Ghidra.