I am working on reverse engineering an algorithm and I am using IDA. This is how the stack definition looks:
var_41E= word ptr -41Eh
s= byte ptr -412h
var_401= byte ptr -401h (v23)
var_400= byte ptr -400h (v22)- IDA shows as db 835
var_BD= word ptr -0BDh
var_13= byte ptr -13h (v25)
var_12= byte ptr -12h
var_11= byte ptr -11h
var_10= byte ptr -10h
var_F= byte ptr -0Fh
var_E= byte ptr -0Eh
var_D= byte ptr -0Dh
arg_0= dword ptr 8
dest= dword ptr 0Ch
arg_8= dword ptr 10h
There is this code block:
loc_1B41AA:
mov al, [ebp+edx+var_12]
xor [ebp+edx+var_400], al
inc edx
cmp edx, 4
jnz short loc_1B41AA
Which translates to:
do
{
v23[v11] ^= *(&v26 + v11);
++v11;
}
while ( v11 != 4 );
The problem I'm having is, this v23 is never used. I was thinking perhaps this is a pointer to another spot in memory, but I'm unable to find this.
The next section of code reads:
do
{
*(&v25 + v11) ^= *(&v22 + v11);
++v11;
}
while ( v11 != 5 );
Which again, v22 is never used anywhere else. Nor is v25.
I feel like I'm missing something obvious. I have been able to reversed previous formula's so I just don't understand why this seems to allude me. I was thinking maybe this is some trash code to throw people off, but I'm not sure because I've replicated the rest of the process and the results aren't right. If I need to provide additional information, let me know.
Here is the entire code block for better reference: v9 is a 8 byte buffer. byte 0 is not used by the algorithm. I see "*(&v25 + v11) ^= *(&v22 + v11)" which seems like it could be altering this array? I suck at stack related things...
v26 = *(_BYTE *)(v9 + 1) ^ 0x42;
v27 = *(_BYTE *)(v9 + 2) ^ 0x4F;
v28 = *(_BYTE *)(v9 + 3) ^ 0x4C;
v29 = *(_BYTE *)(v9 + 4) ^ 0x37;
v30 = *(_BYTE *)(v9 + 5) ^ 0x37;
v10 = *(_BYTE *)(v9 + 6);
v11 = 0;
v31 = v10 ^ 0x36;
do
{
v23[v11] ^= *(&v26 + v11);
++v11;
}
while ( v11 != 4 );
LOBYTE(v11) = 1;
do
{
*(&v25 + v11) ^= *(&v22 + v11);
++v11;
}
while ( v11 != 5 );
.text:001B4166 mov esi, [ebp+arg_0]
.text:001B4169 mov edx, [esi+120h]
.text:001B416F add esp, 10h
.text:001B4172 mov al, [edx+1]
.text:001B4175 xor eax, 42h
.text:001B4178 mov [ebp+var_12], al
.text:001B417B mov al, [edx+2]
.text:001B417E xor eax, 4Fh
.text:001B4181 mov [ebp+var_11], al
.text:001B4184 mov al, [edx+3]
.text:001B4187 xor eax, 4Ch
.text:001B418A mov [ebp+var_10], al
.text:001B418D mov al, [edx+4]
.text:001B4190 xor eax, 37h
.text:001B4193 mov [ebp+var_F], al
.text:001B4196 mov al, [edx+5]
.text:001B4199 xor eax, 37h
.text:001B419C mov [ebp+var_E], al
.text:001B419F mov al, [edx+6]
.text:001B41A2 xor edx, edx
.text:001B41A4 xor eax, 36h
.text:001B41A7 mov [ebp+var_D], al
.text:001B41AA
.text:001B41AA loc_1B41AA: ; CODE XREF: USBIO::ReadDS1995KeyData(uchar *,ushort)+243j
.text:001B41AA mov al, [ebp+edx+var_12]
.text:001B41AE xor [ebp+edx+var_400], al
.text:001B41B5 inc edx
.text:001B41B6 cmp edx, 4
.text:001B41B9 jnz short loc_1B41AA
.text:001B41BB mov dl, 1
.text:001B41BD
.text:001B41BD loc_1B41BD: ; CODE XREF: USBIO::ReadDS1995KeyData(uchar *,ushort)+256j
.text:001B41BD mov al, [ebp+edx+var_401]
.text:001B41C4 xor [ebp+edx+var_13], al
.text:001B41C8 inc edx
.text:001B41C9 cmp edx, 5
.text:001B41CC jnz short loc_1B41BD
.text:001B41CE xor ecx, ecx
.text:001B41D0
.text:001B41D0 loc_1B41D0: ; CODE XREF: USBIO::ReadDS1995KeyData(uchar *,ushort)+283j
.text:001B41D0 cmp ecx, 11h
.text:001B41D3 jbe short loc_1B41DA
.text:001B41D5 cmp ecx, 15h
.text:001B41D8 jbe short loc_1B41F2
.text:001B41DA
.text:001B41DA loc_1B41DA: ; CODE XREF: USBIO::ReadDS1995KeyData(uchar *,ushort)+25Dj
.text:001B41DA mov edx, 6
.text:001B41DF mov eax, ecx
.text:001B41E1 mov esi, edx
.text:001B41E3 xor edx, edx
.text:001B41E5 div esi
.text:001B41E7 mov al, [ebp+edx+var_12]
.text:001B41EB xor [ebp+ecx+s], al
.text:001B41F2
.text:001B41F2 loc_1B41F2: ; CODE XREF: USBIO::ReadDS1995KeyData(uchar *,ushort)+262j
.text:001B41F2 inc ecx
.text:001B41F3 cmp ecx, 400h
.text:001B41F9 jnz short loc_1B41D0
.text:001B41FB xor esi, esi
.text:001B41FD lea ecx, [ebp+s]
.text:001B4203 xor edx, edx