How should I approach reverse engineering this text encoding?

Question

So I'm trying to hack the translation from the PS4 version of a game into the Vita version. The script files were conveniently uncompressed, and I was able to drop them in and have it working without a hitch - great!

However, various other message files and quest summaries and the like are not so convenient.

Here's a comparison of two files in a hex editor:

At first I thought it was a simple byte-pair compression, but there's obviously more to it than that, because if you look at the places which correspond to "Let's go!" and "Let's do this!" they don't start with the same string of characters at all.

I uploaded the PS4/Vita versions of a file with rather more readable text: https://www.dropbox.com/s/bw0nvexyi9ww2be/hm%20vita?dl=0 https://www.dropbox.com/s/sk74zadvndc8v9t/hm%20ps4?dl=0

Going through it and looking for common and recurring words, I found this:

Goblin Thief
´0Ö0ê0ó0·0ü0Õ0
B430 D630 EA30 F330 B730 FC30 D530

Goblin Thief Archer
´0Ö0ê0ó0·0ü0Õ0¢0ü0Á0ã0ü0
B430 D630 EA30 F330 B730 FC30 D530 A230 FC30 C130 E330 FC30

Ancient Grief
¨0ó0·0§0ó0È0°0ê0ü0Õ0
A830 F330 B730 A730 F330 C830 B030 EA30 FC30 D530

Grief Screamer
°0ê0ü0Õ0¹0¯0ê0ü0Þ0ü0
B030 EA30 FC30 D530 B930 AF30 EA30 FC30 DE30 FC30

Grief
EA30 FC30 D530

Thief
B730 FC30 D530

So you can see that FC30 D530 is "ief". But then I look for more occurences of "gr"

Deep Grudge
Ç0£0ü0×0°0é0Ã0¸0
C730 A330 FC30 D730 B030 E930 C330 B830

And you don't see the EA30 that starts off "Grief".

I have a feeling FC30 could be some kind of switch byte, either an upper case indication or possibly marking the use of some kind of lookup table? It's also interesting that all the lines which are just objectives/boss names have the --30 structure, but some of the descriptive passages don't seem to.

The additional problem, of course, is that the uncompressed English text from the PS4 version won't be a 100% perfect match for the text from the Vita version -- that's the whole point, after all! So even when I look at the name "Deep Grudge" and notice that I don't see anything which looks like it would correspond to the "Gr" from "Grief", I can't be certain that they didn't change the name in the PS4 version.

Does anyone have any suggestions on how I should be approaching this? If I'm right and there's actually some kind of compressed lookup table business going on, then it might be effectively impossible to reverse, right?

Answer 1

The way you can learn this encoding is to study Japanese. :) From the first line of your text file diff, we can see that this on the left:

1131 3210 3130 3330 1021 0000 0000 0000  .12.1030.!......

Translates to this on the right:

1100 3100 3200 1000 3100 3000 3300 3000  ..1.2....1.0.3.0.
1000 01ff 0000 0000 0000 0000 0000 0000  .................

This is a very strong hint that the file on the left is using an 8-bit encoding and that the one on the right is using a 16-bit encoding. The digits are translated in a very straightforward way, but the exclamation point ("!") is 0x21 in ASCII or UTF-8 but 0x01ff is the Unicode "fullwidth exclamation mark" encoded as UCS-2 (that is, UTF-16 encoded as little-endian).

So with the hint about encoding we can see that the hex you've identified as "Goblin Thief"

b430 d630 ea30 f330 b730 fc30 d530

is rendered as ゴブリンシーフ which is the Katakana representation of the English words "Goblin Thief". It's common for Japanese speakers to render foreign words into phonetic equivalents with Katakana. So taking it syllable-by-syllable we have:

ゴ  go
ブ  bu
リ  ri
ン  n
シ  shi
ー  (extended vowel)
フ  fu

So if we say it aloud, "go bu ri n shi-i fu" sounds like "Goblin Thief" as pronounced by a native Japanese speaker. You can try Google translate to experiment a bit more and to see and hear what this sounds like.