Here are some thoughts on the fundamental problem and a possible solution; even if the full system goes way beyond your dev budget, some key ideas might still be useful for fashioning your own solution.
Crypto is of little use if you don't have the leverage that makes the crypto algorithm itself the weakest link in the system, just like a ten-inch steel door won't offer security if it is installed in a tent or in a knee-high fence. If the AES implementation is external to your bytecode blob then its entry point is easy to hook, serving the key up on a silver platter.
There are several factors working against you here.
(1) Modularisation greatly aids analysis. Once a body of code has been recognised as serving a certain function - perhaps even a well-known algorithm like CRC32, MD5 or AES - the purpose of its invocations is clear.
(2) Optimising compilers greatly aid analysis, by reducing redundancy. Unless they're overly fond of inlining.
(3) All external interactions of your code (OS calls, library calls) give the attacker a clue as to what's going on.
Here's how you can win anyway: you need to institute a massive shell game to render those clues worthless, you need to break modularisation by interleaving a great number of computations; further, you need to make the state to be verified part of the key, and you need to bind the whole shebang to a particular game session, client process and machine.
One way of doing this is to write a kind of byte code interpreter that implements the fundamental operations you need for the verification: xor two cells, add two others, hash a string, checksum a memory range, retrieve the process id and so on. Essentially a tiny general script similar to IDC or NWScript, with a couple of special 'macro' ops thrown in. Elementary operations take operands from an array of cells (registers) and deliver their results to some cell in the array. Every cycle (operation) also involves cycling a set of simple RNGs (xorshift, KISS, whatever) and a shuffle step of the function table and cell array based on those RNGs. The critical point is that the elements of the state to be verified must be reducible to numeric values that can be used to perturb the RNG state as well.
As soon as a single step of the verification returns the wrong result, all subsequent instructions will be decoded erroneously, meaning the code cannot even be analysed in an environment that fails verification. This protects everything that's computed after the end of the verification code proper, like a session key for example.
All that remains is binding every bytecode blob to a particular client session, that is, to generate a blob specifically for it. Another client session gets a different blob, so there's no info that can be ripped from a legit client session and then used to unlock a bot session.
The verification needs to be broken down into a series of small steps; i.e. instead of checksumming one big memory range in a single go, do dozens of smaller checksum operations.
You'll be writing the verification code in your simple script language as usual; a special program analyses the dependency graph of the resulting object code and determines which bytecode can be executed when, effectively inlining everything into one huge monster function. That's the preprocessing step, done once. Then follows a code generation step where schedulable bytecodes are chosen in a pseudo-random fashion (based on a session key) and compiled into a blob. As each bytecode is chosen, it must also be executed in an environment that passes muster, for proper updating of the interpreter state (RNG state, shuffle state of function table and cell array). There can be hundreds of separate computation strands that can be schedulable at a given point, at least early on.
In addition to controlling bytecode scheduling, the session key can also control the verification operations that you compile into a particular session blob (perhaps choosing from a pool, and parametrising the elementary checksum ranges). This way, analysing a legit client's blob gives the attacker only a fraction of the verification operations that they have to fool/emulate in order to succeed. In extremis you can get away with checksumming only a tiny fraction of the process image while your attackers need to present (or virtualise) the whole image in a pristine fashion, because they don't know which particular areas will be checksummed.
Implementing the bytecode system on top of another bytecode system may constrain things a bit, mostly in terms of performance (achievable complexity) and achievable obscurity (recognisable function call to retrieve the process id instead of plain, unnamed machine instructions peeking
gs:[30h] and going from there). But the principle holds, even if you have to use a giant switch statement instead of a function table. For some strange reason, it seems to be somewhat customary to implement crypto operations (RNGs like Marsaglia's KISS, hashes, ciphers) on top of the bytecode system... Probably because it's so easy to write and so hard to analyse.
Leverage - the way this integrates into the overall system - is just as important here as with crypto. You don't want your ten-inch steel door to be bypassed.
Also, systems like this are not resilient at all, they are very brittle. If one single verification op is too eager/constraining then you'll have thousands of legitimate clients howling in outrage...
The problem here is that things like anti-malware systems, graphics drivers and so on like to f*ck things up and inject DLLs into running processes, which can make white-listing loaded modules a bit tricky. For example, an attacker may inject a DLL with the NVIDIA name into a game running on an ATI card. Detouring is quite common as well.
Note: canned solutions like automatic obfuscators tend to fall to canned hacks. Whether that's a problem or not depends more on accident (presence of skilled reversers among your audience, and the amount of time they're willing to invest) than anything else. For example, there don't seem to be any generic unpackers for the EA/Denuvo sh*te out there even though their schemes are about a decade behind the state of the art. I guess there was no one who had both the skill and a thirst for fame...