Recognize a decryption algorithm

Question

I'm trying to reverse a malware that builds its IAT at runtime. Due to my inexperience, I'm having trouble to understand this function that accepts into EAX a dword (maybe some sort of hash) and into EDX the base address of kernel32.dll. Could you point me how can I work it out? I can't use the decompiler right now.

Answer 1

The function parses the PE header to locate the IMAGE_EXPORT_DIRECTORY which has the structure

NumberOfNames contains the number of symbols exported by this PE and is located at an offset of 0x18.

AddressOfNames is a pointer to an array of null-separated list of exported function names. This is located at offset 0x20.

Using the NumberOfNames value it iterates over the list of exported function names and calculates a hash value for each.

The algorithm to calculate hash is something like.

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

int main()
{
    // The name to hash
    char name[] = "GetModuleFileNameA";

    unsigned int hash = 0;
    unsigned char ch, cl;

    for (int i=0; i<strlen(name); i++)
    {
        ch = ((hash >> 8) & 0xFF) ^ name[i];
        hash = (hash & 0xffff00ff) | (ch << 8);
        hash = _rotl(hash, 8);        
        cl =  (hash & 0xFF) ^ ((hash >> 8) & 0xFF);
        hash = (hash & 0xFFFFFF00) | cl;
    }
    printf("%08X", hash);
}

If the calculated hash matches, it returns the corresponding address of the API.

The above code calculates the hash of GetModuleFileNameA which comes out to 416F346F. The code can thus be assumed to be correct.

Check here: https://rextester.com/NIBW6473