How can I identify how this string is being encoded so I can replicate it?

Question

A friends company uses an old VB6 application to generate encoded strings to then use in their also old and out of date database system.

They have the application running in standalone mode so I can test inputs and have it generate an encoded output, but they wanted to migrate it into a web based system to build the request string to prevent someone from having to access the old desktop locally.

Input: AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz
Output: ~^}]|\sSrRqQpPwWvVuUtTkKjJiIhHoOnNmMlLcCbBaA`@gGfFeE

Input: 1234567890
Output: 4=<3210765

Input: 1
Output: 5

Input: 12
Output: 65

Input: 123
Output: 765

Any ideas on where I can start to figure this out? It looks like it does a single character encode and then reverses the string.

EDIT: Gets a little more complex it seems.... If the input cross into a new column it appears to subtract 4?

Input: ?
Output: ;

Input: 789:;<=>?
Output: ;:98?>=<3

Looks like maybe the application is using multiple definitions? Or doing some type of math range to determine to add or subtract?

Decompiled Code that makes very little sense to me.

Private Sub CommandButton_Click() '408C90
  Dim var_44 As TextBox
  Dim var_48 As TextBox
  loc_00408CB5: var_8 = &H4010C0
  loc_00408D16: Set var_44 = Me
  loc_00408D25: var_28 = password.Text
  loc_00408D6D: Set var_44 = Len(var_28)
  loc_00408D7C: var_28 = password.Text
  loc_00408DBE: 
  loc_00408DC5: If Len(var_28) < 0 Then GoTo loc_00408EDA
  loc_00408DE5: var_54 = Me
  loc_00408DF2: var_64 = 1
  loc_00408DF9: var_6C = 2
  loc_00408E00: var_44 = 0
  loc_00408E07: var_5C = 9
  loc_00408E1C: var_28 = CStr(Mid$(vbObject, Len(var_28), 1))
  loc_00408E34: call __vbaStrI2(Asc(var_28) xor eax, Me, var_28, var_44, 0040856Ch, 000000A0h, 000000A0h, 000000A0h, 000000A0h)
  loc_00408E70: var_84 = var_24
  loc_00408E76: var_8C = 8
  loc_00408E81: Var_Ret_1 = CLng(__vbaStrI2(Asc(var_28) xor eax, Me, var_28, var_44, 0040856Ch, 000000A0h, 000000A0h, 000000A0h, 000000A0h))
  loc_00408E8C: var_5C = Chr(Var_Ret_1)
  loc_00408EA1: var_6C = var_24 & var_24
  loc_00408EC5: eax = var_5C Or FFFFFFFFh
  loc_00408ECB: var_5C Or FFFFFFFFh = var_5C Or FFFFFFFFh + Len(var_28)
  loc_00408ED5: GoTo loc_00408DBE
  loc_00408EDA: 
  loc_00408EF0: var_A0 = vbEmpty
  loc_00408F04: Set var_44 = Me
  loc_00408F0F: var_28 = ip.Text
  loc_00408F3B: Set var_48 = var_28
  loc_00408F4A: var_30 = User.Text
  loc_00408FC0: var_40 = "http://" & var_28 & "/_common/servlet/lvl5/login?language=en&name=" & var_30 & "&pass=" & var_24
  loc_00408FCA: Me.MousePointer = var_40
  loc_0040902F: GoTo loc_00409081
  loc_00409080: Exit Sub
  loc_00409081: 
  loc_00409091: Exit Sub
End Sub

Answer 1

Edit: The encryption is Bitwise XOR with the key 0x04 (see the bottom of this answer)

The application uses a simple Substitution Cipher (Or, to be exact Caesar Cipher with shift 4) and then perform reverse() on the function.

We can domnstrate it using python's maketrans method:

The method maketrans() returns a translation table that maps each character in the intabstring into the character at the same position in the outtab string. Then this table is passed to the translate() function.

Here's a quick script for example that proves this is the encryption used:

from string import maketrans

# Define list of characters
chars = "AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz1234567890"  

# Perform Caesar Cipher on our string
cipher_chars = ""
for c in chars:
    cipher_chars+= chr(ord(c)+4)

# Create a translation table
transition = maketrans(chars, cipher_chars)

# We use "[::-1]" to reverse the string
def encrypt (plaintext):
    return plaintext.translate(transition)[::-1] 

Let's try it:

encrypt("1")
'5'

encrypt ("12")
'65'

encrypt("123")
'765'

And it also works vise versa:

encrypt("65")
'12'

encrypt("765")
'123'

Edit:

According to the decompiled code you've added and @Ilmari's great comment and response which was correctly pointed out the following finding: the encryption is XOR with "0x04".

Here's the updated encrypt function:

def encrypt (plaintext):
    return ''.join(chr(ord(c) ^ 4) for c in plaintext)[::-1]

Answer 2

Based on your examples, it looks like the strings are reversed and each byte is XORed with the byte 4. Here's a quick Perl one-liner to demonstrate this:

perl -lne 'print reverse($_) ^ ("\x04" x length($_))'

and what happens when you apply it to your test strings:

$ cat test.txt
AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz
1234567890
1
2
123
?
789:;<=>?

$ perl -lne 'print reverse($_) ^ ("\x04" x length($_))' test.txt 
~^}]|\sSrRqQpPwWvVuUtTkKjJiIhHoOnNmMlLcCbBaA`@gGfFeE
4=<3210765
5
6
765
;
;:98?>=<3

The fact that the input string gets reversed is pretty obvious if you compare the encryptions of 1 → 5 and 2 → 6 with 123 → 765.

After figuring that out, and guessing that the rest is just a bytewise substitution cipher, you can figure out the pattern to the substitutions by looking at the encryptions of the letters. Specifically, we can see that (ignoring the reversal), the letters:

ABCDEFGHIJKLMNOPQRSTUVWXYZ

encrypt to:

EFG@ABCLMNOHIJKTUVWPQRS\]^

Noting that @ comes right before A in ASCII (and ` likewise comes right before a), we can see that the encrypted alphabet is shuffled around in chunks of 4 letters:

Input:  ABC DEFG HIJK LMNO PQRS TUVW XYZ
Output: EFG @ABC LMNO HIJK TUVW PQRS \]^

We can also see that the mapping is self-inverse: HIJK encrypts to LMNO while LMNO encrypts back to HIJK. This, along with the regular pattern of letter position swaps, should be a pretty strong hint that we might be dealing with bitwise XOR. Comparing the ASCII codes of any pair of plaintext/ciphertext letters is then sufficient to reveal the constant key byte (which we can then test on other known input strings to confirm our guess).