Let us suppose that I have a program that perform some well known cryptographic routines like AES, RSA or whatever. I would like to detect when such algorithms run, on which key material and possibly on which input and output data streams.
The idea is to run the whole program inside a virtual machine and then instrument the virtual machine itself to study the behavior of the program. For example, I might implement heuristics to check whether AES S-box is computed (when you see certain bytes and some clock cycles later the corresponding bytes in S-box show up in some register, then you might have a clue that S-box is being run). Tracing down how those bytes were moved across memory before might lead me to where the encryption key is stored.
The point of doing all of this in a virtual machine is that I can make up the environment so that the program is unable to detect that I am trying to spy on him. This way I would like to avoid all the mess with protectors and whatever and just see how it behaves.
Are there already attempts at reverse engineering programs in a similar way? By "similar way" I mean running the program in a virtual machine and looking how it moves data around the memory and on the registers, in order to find patterns that may help me to track down the information I am seeking.
In case it helps, the architecture I would like to use is x86 and the practical program is the one that I described above: interception of known cryptography procedure.