Simplifying and reversing an encryption/decryption algorithm in ASM X86 / Python

Question

I'm trying to reverse a decoding function in order to produce the function for encoding (for the curious this is a save game for a legacy adventure game which use codes for saving).

So far I have ported the decoding function from ASM X86 to Python and it's working fine. My problem is I'm not able to reverse the function because I don't understand what the code does with the bitwise operations. Specially the bitwise OR is like voodoo to me.

I'd like some help to simplify/understand the function and reverse it.

My code so far:

def load_and_decode_value(text, index, table):
  current_char = ord(text[index])
  return table[current_char]

def text_to_hex(text):
  with open("DECODE_TABLE.bin", mode='rb') as file:
    DECODE_TABLE = file.read()

  decoded = []
  char_index = 0
  text_len = len(text)

  for i in range(0, 5):

    val_0 = load_and_decode_value(text, char_index, DECODE_TABLE)
    char_index += 1
    val_1 = load_and_decode_value(text, char_index, DECODE_TABLE)
    char_index += 1
    val_next_0 = (val_0 * 2048) | (val_1 * 64)
    val_1 = (val_next_0 // 256) & 0xFF
    decoded.append(val_1)

    if i == 4:
      break

    val_2 = load_and_decode_value(text, char_index, DECODE_TABLE)
    char_index += 1
    val_3 = load_and_decode_value(text, char_index, DECODE_TABLE)
    char_index += 1
    val_next_1 = (((val_2 * 2) | val_next_0) * 16777216) | (val_3 * 1048576)
    val_3 = (val_next_1 // 16777216) & 0xFF
    decoded.append(val_3)

    val_4 = load_and_decode_value(text, char_index, DECODE_TABLE)
    char_index += 1
    val_next_2 = val_next_1 | (val_4 * 32768)
    val_4 = (val_next_2 // 65536) & 0xFF
    decoded.append(val_4)

    val_5 = load_and_decode_value(text, char_index, DECODE_TABLE)
    char_index += 1
    val_6 = load_and_decode_value(text, char_index, DECODE_TABLE)
    char_index += 1
    val_next_2 = val_next_2 | (val_5 * 1024)
    val_next_3 = val_next_2 | (val_6 * 32)
    val_6 = (val_next_3 // 256) & 0xFF
    decoded.append(val_6)

    val_7 = load_and_decode_value(text, char_index, DECODE_TABLE)
    char_index += 1
    val_8 = (val_next_3 | val_7) & 0xFF 
    decoded.append(val_8)

  return decoded

The content of the DECODE_TABLE.bin file:

FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FF FF 00 01 02 03 04 05 06 07 08 09 FF FF FF FF FF FF FF 0A 0B 0C 0D 
0E 0F 10 11 01 12 13 01 14 15 00 16 17 18 19 1A FF 1B 1C 1D 1E 1F FF 
FF FF FF FF FF 0A 0B 0C 0D 0E 0F 10 11 01 12 13 01 14 15 00 16 17 18 
19 1A 1B FF 1C 1D 1E 1F FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
FF FF FF                                                             

Answer 1

Tip is Comment every line.

Ex)

while i < HARDCODED_LIMIT:
    # Break when text index is out of range
    if text_char_index >= text_len:
        break
    # Load text's ascii number
    eax = ord(text[text_char_index])
    # Increase text index
    text_char_index += 1
    # Load encode array's Nth data (if indexth text's ascii is 116, load 26)
    eax = TEXT_ENCODE_ARRAY[eax]
    # If Nth char is not a character(0~9a~zA~Z), then break
    if eax < 0: # This code is wrong, (0xff < 0) is False
        break

Then try to make code in high-level language

text: str = "Hello"
text.replace("t", "\x1a")

In this level, unusable code is snipped such as

esi = eax & 0xFF # eax is smaller than 0xff (eax is Nth data of ENCODE_ARRAY.bin), so this code is same as esi = eax

---

Change bitwise operator to human friendly operator

esi = esi << 0xB # This could be
esi = esi * 2048 # and, this could be
esi *= 2048

---

Try to change common function to def func1()

I think, it seems that the above is repeated

Just make a function code and fill in the changing part with parameters

Then you can read it easily, you can change it to asm code, make decode function, make decode asm code.

PS. If you don't understand bitwise operators, leave them as bitwise operator.