I recently came across a script that would read a file into a string, then do the equivalent of
CallWindowProc((pointer to string), (pointer to encrypted string), (length of encrypted string), 0, 0)
in the scripting language to decrypt an encrypted string defined earlier in the file. What's going on here? How did the call to CallWindowProc decrypt the string?
Normally, the first parameter to CallWindowProc would be a function pointer. CallWindowProc treats the pointer to the string as a function pointer, and calls it. As long as the heap that the string exists in is executable, it will be executed as if it were machine code. The other four parameters passed to CallWindowProc are also passed to the function pointer, and can be used by the machine code however it's designed. In this case, it looks like the machine code your script is loading is a decryption subroutine. Try looking at the machine code in a disassembler.
