Decompilers points to non-existing virtual function

Question

As part of solving the Hidden password challenge, I found an condition calls a virtual function

the v14 points to v8 variable :

and the functions in the program does not make sense for me, there is no two-args functions in binary program, does this bytes mean something such as signatures/evaluable code/etc..

  v8[0] = 0x28BF16683619A05BLL;
  v8[1] = 0x4DD3CE3A2552E799LL;
  v8[2] = 0xA5ED9BE182304449LL;
  v8[3] = 0x6E27E1473B191037LL;
  v8[4] = 0x6DA9EC4E7AC0DAECLL;
  v8[5] = 0x8929723C31C59039LL;
  v8[6] = 0xEA92AC15DE3C3F69LL;
  v8[7] = 0x828DD2F713F6E8BELL;
  v8[8] = 0xBB4D607B1C553C6FLL;
  v8[9] = 0x7DC2D2F3EC43EF5BLL;
  v8[10] = 0x4DAF64150084DC96LL;
  v8[11] = 0xE1F1361E21C67AB9LL;
  v8[12] = 0xA4B498C90BE95F82LL;
  v8[13] = 0xB439B94451F266B5LL;
  v8[14] = 0x2380C814A4F0145BLL;
  v8[15] = 0x808581A5B7FB9D7ELL;
  v8[16] = 0x589B2B23881C5633LL;
  v8[17] = 0xBBAA188D8CDE35D8LL;
  v8[18] = 0xF6F8FD3AEB6DD0D2LL;
  v8[19] = 0x2FEAA6B6AE8530B8LL;
  v8[20] = 0xB30EDC56B009E85FLL;
  v8[21] = 0xFBD9E747FFC36C8FLL;
  v8[22] = 0x18194F7045E8F66LL;
  v8[23] = 0xC27B4D434FE8BEEALL;

code = [
    0x28,0xBF,0x16,0x68,0x36,0x19,0xA0,0x5B,
    0x4D,0xD3,0xCE,0x3A,0x25,0x52,0xE7,0x99,
    0xA5,0xED,0x9B,0xE1,0x82,0x30,0x44,0x49,
    0x6E,0x27,0xE1,0x47,0x3B,0x19,0x10,0x37,
    0x6D,0xA9,0xEC,0x4E,0x7A,0xC0,0xDA,0xEC,
    0x89,0x29,0x72,0x3C,0x31,0xC5,0x90,0x39,
    0xEA,0x92,0xAC,0x15,0xDE,0x3C,0x3F,0x69,
    0x82,0x8D,0xD2,0xF7,0x13,0xF6,0xE8,0xBE,
    0xBB,0x4D,0x60,0x7B,0x1C,0x55,0x3C,0x6F,
    0x7D,0xC2,0xD2,0xF3,0xEC,0x43,0xEF,0x5B,
    0x4D,0xAF,0x64,0x15,0x00,0x84,0xDC,0x96,
    0xE1,0xF1,0x36,0x1E,0x21,0xC6,0x7A,0xB9,
    0xA4,0xB4,0x98,0xC9,0x0B,0xE9,0x5F,0x82,
    0xB4,0x39,0xB9,0x44,0x51,0xF2,0x66,0xB5,
    0x23,0x80,0xC8,0x14,0xA4,0xF0,0x14,0x5B,
    0x80,0x85,0x81,0xA5,0xB7,0xFB,0x9D,0x7E,
    0x58,0x9B,0x2B,0x23,0x88,0x1C,0x56,0x33,
    0xBB,0xAA,0x18,0x8D,0x8C,0xDE,0x35,0xD8,
    0xF6,0xF8,0xFD,0x3A,0xEB,0x6D,0xD0,0xD2,
    0x2F,0xEA,0xA6,0xB6,0xAE,0x85,0x30,0xB8,
    0xB3,0x0E,0xDC,0x56,0xB0,0x09,0xE8,0x5F,
    0xFB,0xD9,0xE7,0x47,0xFF,0xC3,0x6C,0x8F,
    0x18,0x19,0x4F,0x70,0x45,0xE8,0xF6,0x6,
    0xC2,0x7B,0x4D,0x43,0x4F,0xE8,0xBE,0xEA,
    0xcb,0x87,0xce,0xb3
]

and the XOR part *((_BYTE *)v8 + (int)k) ^= v11; XOR with the key generated by other function

unsigned __int64 sub_1169()
{
  // qword_4040 -> 0x1
  qword_4040 = 1103515245 * qword_4040 + 12345;
  return ((unsigned __int64)qword_4040 >> 16) & 0x7FFF;
}

outputs 0x41c6 and Xor'ing that key It won't give anything understandable and even useful

code = [
    0x28,0xBF,0x16,0x68,0x36,0x19,0xA0,0x5B,
    0x4D,0xD3,0xCE,0x3A,0x25,0x52,0xE7,0x99,
    0xA5,0xED,0x9B,0xE1,0x82,0x30,0x44,0x49,
    0x6E,0x27,0xE1,0x47,0x3B,0x19,0x10,0x37,
    0x6D,0xA9,0xEC,0x4E,0x7A,0xC0,0xDA,0xEC,
    0x89,0x29,0x72,0x3C,0x31,0xC5,0x90,0x39,
    0xEA,0x92,0xAC,0x15,0xDE,0x3C,0x3F,0x69,
    0x82,0x8D,0xD2,0xF7,0x13,0xF6,0xE8,0xBE,
    0xBB,0x4D,0x60,0x7B,0x1C,0x55,0x3C,0x6F,
    0x7D,0xC2,0xD2,0xF3,0xEC,0x43,0xEF,0x5B,
    0x4D,0xAF,0x64,0x15,0x00,0x84,0xDC,0x96,
    0xE1,0xF1,0x36,0x1E,0x21,0xC6,0x7A,0xB9,
    0xA4,0xB4,0x98,0xC9,0x0B,0xE9,0x5F,0x82,
    0xB4,0x39,0xB9,0x44,0x51,0xF2,0x66,0xB5,
    0x23,0x80,0xC8,0x14,0xA4,0xF0,0x14,0x5B,
    0x80,0x85,0x81,0xA5,0xB7,0xFB,0x9D,0x7E,
    0x58,0x9B,0x2B,0x23,0x88,0x1C,0x56,0x33,
    0xBB,0xAA,0x18,0x8D,0x8C,0xDE,0x35,0xD8,
    0xF6,0xF8,0xFD,0x3A,0xEB,0x6D,0xD0,0xD2,
    0x2F,0xEA,0xA6,0xB6,0xAE,0x85,0x30,0xB8,
    0xB3,0x0E,0xDC,0x56,0xB0,0x09,0xE8,0x5F,
    0xFB,0xD9,0xE7,0x47,0xFF,0xC3,0x6C,0x8F,
    0x18,0x19,0x4F,0x70,0x45,0xE8,0xF6,0x6,
    0xC2,0x7B,0x4D,0x43,0x4F,0xE8,0xBE,0xEA,
    0xcb,0x87,0xce,0xb3
]

key = [0x41,0xc6]*98

code = ""

for i in range(196):
    code+=chr(code[i]^key[i])

what is this line of code really does in this context :

if ( ((unsigned int (__fastcall *)(char *, size_t))v14)(a2[1], v5) )

Answer 1

sub_1169 is actually an LCG based random number generator. sub_1155 sets the seed value for this LCG. So sub_1155 is similar to srand and sub_1169 is rand

Because of this the xor key is not the same for every byte.

srand(seed);
for ( k = 0; k <= 196; ++k )
{
  v9 = rand();
  v8[k] ^= v9;
}

Now

v14 = (int (__fastcall *)(char *, int))v8;

This means we are assigning byte data pointed by v8 as function or code - specifically a function that takes a string and an int as a param and returns an int. Something like

bool verify_flag(char *flag, int len);

So v8 is some sort of shellcode which is decoded in the xor fashion as above. Later that function is called to verify argv[1]

if ( v12(argv[1], v5) )
  puts("Good password!");
else
  puts("Invalid password!");

The first step of getting the shellcode decoded correctly is to figure out the input that calculates some sort of "checksum" which when used as seed to srand will decode to perfect x86_64 code. This is the "checksum" logic.

seed = 0x7FFFFFFF;
for ( i = 0; ; ++i )
{
    v3 = i;
    if ( v3 >= strlen(argv[1]) )
    break;
    seed += i * argv[1][i];
}

Additionally the binary sets up some mod based constraints that will help you figure out the correct checksum using z3.

from z3 import *
# https://reverseengineering.stackexchange.com/questions/30303/z3-is-unable-to-predict-the-operand
v7 = [123, 456, 789, 987, 654, 321]
v6 = [92, 29, 380, 2, 497, 296]
arrl = 14
argv1 = [BitVec(f'a{i}', 32) for i in range(arrl)]

solver = Solver()
v18 = BitVecVal(0x7fffffff, 32)

for i in range(arrl):
    solver.add(argv1[i] < 128)
    solver.add(argv1[i] > 32)
    v18 += i*argv1[i]

for i in range(6):
    solver.add(URem(v18, v7[i]) == v6[i])

print(solver.check())
print("".join(map(chr, [solver.model()[argv1[i]].as_long()
      for i in range(arrl)])))

Once you know an input that will produce the correct seed for srand such that rand generates the correct order of values for v8 shellcode to be decoded correctly.

You can then run and dump this decoded shellcode or xor it statically with correct rand values.

This shellcode similarly can be analysed with IDA and modelled with z3. Its a simple xor of 2 buffers and then compare with a static buffer. Unrelated to the answer, I have solved it with angr here