Need some pointers to decode/decrypt/unobfuscate JSON files (updated with related scripts)

Question

I am very new to this whole business so please bear with me.

Goal

My goal is to read JSON data from a game. Since I do not know what kind of information will/will not be useful, here are some of the details:

It is an obscure mobile game by a Japanese company.
All the JSON files were under the folder named binarytableassetsen (they put either en or jp at the end of some of their folders).

Examples

Here's an example of one of the said files called battle_pvp_cost.json.
xxd battle_pvp_cost_2.json > temp.txt

00000000: 1400 0000 6261 7474 6c65 5f70 7670 5f63  ....battle_pvp_c
00000010: 6f73 742e 6a73 6f6e 3701 0000 87ce ef3e  ost.json7......>
00000020: 47d6 93a2 656e a265 6ea2 6a70 ceb4 8e44  G...en.en.jp...D
00000030: ecda 0029 789c 8b56 3254 d251 3202 6263  ...)x..V2T.Q2.bc
00000040: 2036 0162 5320 3603 6273 20b6 0062 4b20   6.bS 6.bs ..bK 
00000050: 3634 0011 864a b10c 00c6 3707 fdce aca5  64...J....7.....
00000060: d333 29ce befb 0353 c3ce 70da a798 a0ce  .3)....S..p.....
00000070: b982 0d5b 00ce 1ab8 1a85 8bce 83dc efb7  ...[............
00000080: 82ce bf39 6750 01ce 1826 94fc 32ce 1ad5  ...9gP...&..2...
00000090: be0d 82ce bf39 6750 02ce 1826 94fc 64ce  .....9gP...&..d.
000000a0: 6dd2 8e9b 82ce bf39 6750 03ce 1826 94fc  m......9gP...&..
000000b0: d100 c8ce f3b6 1b38 82ce bf39 6750 04ce  .......8...9gP..
000000c0: 1826 94fc d101 2cce 84b1 2bae 82ce bf39  .&....,...+....9
000000d0: 6750 05ce 1826 94fc d101 90ce 1db8 7a14  gP...&........z.
000000e0: 82ce bf39 6750 06ce 1826 94fc d101 f4ce  ...9gP...&......
000000f0: 6abf 4a82 82ce bf39 6750 07ce 1826 94fc  j.J....9gP...&..
00000100: d102 58ce fa00 5713 82ce bf39 6750 08ce  ..X...W....9gP..
00000110: 1826 94fc d102 bcce 8d07 6785 82ce bf39  .&........g....9
00000120: 6750 09ce 1826 94fc d103 20ce a15d 25e1  gP...&.... ..]%.
00000130: 82ce bf39 6750 0ace 1826 94fc d103 84ce  ...9gP...&......
00000140: d65a 1577 82ce bf39 6750 0bce 1826 94fc  .Z.w...9gP...&..
00000150: d103 e800 0000 0000                      ........

All of them begins in the same way with a value in the first byte followed by 3 bytes of 0s and then the file name.

Attempts

I used signsrch suggested by this answer but found 0 signatures.
I played around with Burp suite decoder, but the output is equally messed up. The smart decode didn't give any output.
I also found this collection of compression algorithms. Some of them share the characteristic of having some of the data uncompressed like the lzo1x, but none of them seems to have the same amount of special characters. Does this mean there's a layer of obfuscation involved?

If anyone recognises what this is or can point me in the right direction, then I am all ears. Please let me know if there is any more information I can provide to help guide the process. I've tried to put in all the details I think are relevant, but I am new to file decryption and may miss some crucial hints.

Edit

Opening the file correctly in the first place using a text editor that can handle binary data. [Reading 1] [Reading 2] [Reading 3]
The files I got were exported from [Tool 1: DevXUnity] and seems to be different from how I see it on the hex viewer within the tool itself (the first hex dump in this question) just as blabb and w s pointed out. I've included a link to the other version of the first example json. [Reading 1] [Tool 2: UnityEX]
Found some related scripts:

Scripts

Found Parser.Parse(json); in Json.cs and tried to mock it up on Unity but the result is incorrect.

Here're the scripts + json files in a shared gdrive.

Call stacks from when the JSON was first referenced:

battle_pvp_costTableAccessor::constructor()
TableAccessorCore::reset(filename="battle_pvp_cost.json", rootKey="battle_pvp_cost")
TableAccessorCore::reset(filename="battle_pvp_cost.json", rootKey="battle_pvp_cost", isResources=false)
// Branch 1
TableAccessor::get2(name="battle_pvp_cost.json")
TableAccessor::load2(name="battle_pvp_cost.json", isResources=false)
TableResources::load2(name="battle_pvp_cost.json", isResources=false)
TableResources::loadTextAssetFormTableAssets(filename="battle_pvp_cost.json")
TextAsset::ContainsAndLoadU5S("battle_pvp_cost.json")(?)

// Branch 2
Handle::getTable(key="battle_pvp_cost")
uint key2 = CRC32.Compute(str="battle_pvp_cost")
ParameterTableProxy::ParameterTable_table(handle=this.Ptr(?), key=key2)

After brief looking into the files: this is definitely neither encryption nor compression (because of existence of strings and low entropy). I'd call this encoding. — w s, Commented Mar 6, 2018 at 6:35
@IrinThirdwater i added an answer and deleted the comments so doesnt like a prolonged comment session so please edit your question and i will edit my answer if there is some thing to add — blabb, Commented Mar 6, 2018 at 7:01
Are you sure that uploaded files are the same as in your hex dumps ? When I downloading your files all of them are starting with EF BF BD EF BF BD which contradicts to hex dumps you posted in the question. — w s, Commented Mar 6, 2018 at 7:43
Yes, I've noticed this now that both you and blabb pointed it out. I'm trying to understand what's going on at the moment. For now, I've included both versions of battle_pvp_cost.json to the question. — WaterGenie, Commented Mar 6, 2018 at 7:57
FYI: 0xef 0xbf 0xbd is utf-8 character placeholder (see for example this SO question: stackoverflow.com/questions/25222973/weird-characters-in-url ). This probably means that your decoder just trying to present the json files as utf-8 encoded text according to extension - and it looks like that we need this files before the unpacker just as they "seen in the unpacker" as battle_pvp_cost_2.json . — w s, Commented Mar 6, 2018 at 8:26

blabb · Accepted Answer · 2018-03-06 15:49:22Z

you need to post a textual hex-dump using some hex-editor that is capable of writing out

the gnu xxd is capable of doing that in terminal and has been ported to many os

the second image in your question that starts with 0x14 0x00 0x00 0x00 seems

the 0x14 = 20 appears to be the size of filename "battle_pvp_cost.json"

C:\>python
Python 2.7.11 
>>> hex(len("battle_pvp_cost.json"))
'0x14'
>>>

that is followed by 0x137 which appears to be the size of binary data

>>> hex((int(0x153-0x1c)))
'0x137'

this is the data which you seem to be trying to decode / decrypt whatever

87 ce ef 3e ...... until 0xe8

some binary seems to take this blob and writes down the decrypted / decoded /??? version of this blob

you have to find out who uses this blob and may be reverse engineer that binary to understand

the first block-quote seems to be this data the third character is greater than symbol which converts to 0x3e

if you notice the png you posted in your last comment has this pattern interspersed 0xef 0xbf 0xbd for some bytes other bytes are left intact from the second hexdump screen shot

removing the pattern you get what is in the second screen shot

Offset(h) 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000  00 00 00 00 00 00 00 00 00 3E 47 D6 93 00 00 00  .........>GÖ“...
00000010  65 6E 00 00 00 65 6E 00 00 00 6A 70 CE B4 00 00  en...en...jpÎ´..
00000020  00 44 00 00 00 00 00 00 00 29 78 00 00 00 00 00  .D.......)x.....
00000030  00 56 32 54 00 00 00 51 32 02 62 63 20 36 01 62  .V2T...Q2.bc 6.b
00000040  53 20 36 03 62 73 20 00 00 00 00 62 4B 20 36 34  S 6.bs ....bK 64
00000050  00 11 00 00 00 4A 00 00 00 0C 00 00 00 00 37 07  .....J........7.
00000060  00 00 00 CE AC 00 00 00 00 00 00 33 29 CE BE 00  ...Î¬......3)Î¾.
00000070  00 00 03 53 00 00 00 00 00 00 70 DA A7 00 00 00  ...S......pÚ§...
00000080  00 00 00 CE B9 00 00 00 0D 5B 00 00 00 00 1A 00  ...Î¹....[......
00000090  00 00 1A 00 00 00 00 00 00 CE 83 00 00 00 EF B7  .........Îƒ...ï·
000000A0  82 CE BF 39 67 50 01 00 00 00 18 26 00 00 00 00  ‚Î¿9gP.....&....
000000B0  00 00 32 00 00 00 1A D5 BE 0D 00 00 00 CE BF 39  ..2....Õ¾....Î¿9
000000C0  67 50 02 00 00 00 18 26 00 00 00 00 00 00 64 00  gP.....&......d.
000000D0  00 00 6D D2 8E 00 00 00 00 00 00 CE BF 39 67 50  ..mÒŽ......Î¿9gP
000000E0  03 00 00 00 18 26 00 00 00 00 00 00 00 00 00 00  .....&..........
000000F0  00 00 00 00 00 00 00 00 00 1B 38 00 00 00 CE BF  ..........8...Î¿
00000100  39 67 50 04 00 00 00 18 26 00 00 00 00 00 00 00  9gP.....&.......
00000110  00 00 01 2C CE 84 00 00 00 2B 00 00 00 00 00 00  ...,Î„...+......
00000120  CE BF 39 67 50 05 00 00 00 18 26 00 00 00 00 00  Î¿9gP.....&.....
00000130  00 00 00 00 01 00 00 00 00 00 00 1D 00 00 00 7A  ...............z
00000140  14 00 00 00 CE BF 39 67 50 06 00 00 00 18 26 00  ....Î¿9gP.....&.
00000150  00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 6A  ...............j
00000160  00 00 00 4A 00 00 00 00 00 00 CE BF 39 67 50 07  ...J......Î¿9gP.
00000170  00 00 00 18 26 00 00 00 00 00 00 00 00 00 02 58  ....&..........X
00000180  00 00 00 00 00 00 00 57 13 00 00 00 CE BF 39 67  .......W....Î¿9g
00000190  50 08 00 00 00 18 26 00 00 00 00 00 00 00 00 00  P.....&.........
000001A0  02 00 00 00 CE 8D 07 67 00 00 00 00 00 00 CE BF  ....Î..g......Î¿
000001B0  39 67 50 09 00 00 00 18 26 00 00 00 00 00 00 00  9gP.....&.......
000001C0  00 00 03 20 CE A1 5D 25 00 00 00 CE BF 39 67 50  ... Î¡]%...Î¿9gP
000001D0  0A 00 00 00 18 26 00 00 00 00 00 00 00 00 00 03  .....&..........
000001E0  00 00 00 00 00 00 00 00 00 5A 15 77 00 00 00 CE  .........Z.w...Î
000001F0  BF 39 67 50 0B 00 00 00 18 26 00 00 00 00 00 00  ¿9gP.....&......
00000200  00 00 00 03 00 00 00                             .......

those are utf8 replacement charecters

&#65533;&#65533;&#65533;&#65533;&#65533;

��

to answer your comment regarding decoding/decrypting the contents

since you dont have other samples first thing you should do is gather as many samples as you can and confirm if the hypothesis is correct

the hypothesis being first dword = length of file followed by actual file name of said length

on the end of filename another dword that possibly indicates the length of binary blob and followed by actual binary blob of said length

if the file is bigger than the battxx.json does it have other pairs like this

ie another file followed by another binary blob

if the hypothesis pans out

then if you are on windows and you can execute the game or application then you can log the games/ applications activity with procmon / api monitor etc / if you are on nix you have ltrace / strace etc basically a program that monitors activities and logs call stacks / arguments / function reurns etc with that running in background if you executed the game / app and if it accesses this file you can see the call stack and debug the game / app independently and single step around and watch what exactly is happening

a small sample flow diagram would be like Open-> read / map -> manipulate -> write / use

Does this mean the export is irreversible since some characters have been lost? Right now I only have the battle_pvp_cost_2.json as the sole example of the file before the export. I will be able to provide more examples after I have access to my windows machine again if necessary. — WaterGenie, Commented Mar 6, 2018 at 8:43
yes if you have the exported data only then it means all the bytes at replacement characters are lost that said even if you had them you need the application that uses the binary data just posting a correct file (like the last one added to your question) wouldnt be of much help i think because that binary data doesnt appear to be a anything easily identiifable visually you need to find the function in an application which takes this blob and see what it does with it — blabb, Commented Mar 6, 2018 at 8:51
Finally got around to looking at the game files again and this looks like another big set of exercises on its own so I will be marking your answer as accepted for your help and guidance as well as for your patience. If I come across a script that operates on these json files, I will come back and update the questions or post a new one accordingly depending on how related it is by that point. — WaterGenie, Commented Mar 6, 2018 at 14:15
Before we depart from this question, I would like to just clarify this function I should be looking for so that I can actually recognise one when I do come across it/them. If I am understanding things correctly, said functions would be reading one of these json files, and presumably perform some sort of decoding to extract some of its data, right? So my first goal would be to search for any of the names of these json files in the scripts. — WaterGenie, Commented Mar 6, 2018 at 14:19
you have summarized it correct i also edited my post with a short summary so have a look — blabb, Commented Mar 6, 2018 at 15:06

Stack Exchange Network

Need some pointers to decode/decrypt/unobfuscate JSON files (updated with related scripts)

Goal

Examples

Attempts

Scripts

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged
decryption
encodings
or ask your own question.

Linked

Hot Network Questions

Need some pointers to decode/decrypt/unobfuscate JSON files (updated with related scripts)

Goal

Examples

Attempts

Scripts

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged decryptionencodings or ask your own question.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
decryption
encodings
or ask your own question.