Disclaimer: I'm going to expose my workflow when I see similar things. I'm not telling you that this is the best one or the faster, and I'm sure you have a lot of other way to deal with this. But this should give you an insight on how you can do this.
As your question is really wide and does not contain any details (malware name, hash, or what 'failed to analyse' means) my answer is too.
Process Hollowing
Fire-up your favorite debugger, load your binary, and put breakpoints on CreateProcessInternalW, ZwUnmapViewOfsection, WriteProcessMemory and ResumeThread.
This should allow you to observe the full process-hollowing technique.
Hit to CreateProcessInternalW: The process where the injection is going to occur is created by the loader. You should see that the creation flag is set to 0x4 (Suspended). If you open-up procexp or something like that, you should be able to observe the suspended process in your environment.
Hit to ZwUnmapViewOfsection: The previously created process content is wipe. This will make room for the malicious code that is going to be injected later.
Hit to WriteProcessMemory: The malicious code is injected in the empty suspended process. This can be done sections by sections, or from a big blob that contains the full PE. You may want to dump it from here, by looking at the source buffer of the WriteProcessMemory call.
Hit to ResumeThread: The malicious process is then ready to be resume. Your second option is to dump it there, when you are sure that the target process is ready to be launch.
Once you have your dump, start by looking at the sections in PE-bear (or equivalent) to unmappe it. As your dump was taken from memory, the sections are not aligned as it should be on the disk.
Once you re-aligned everything, check if your IAT seems coherent. Even if it use some API hashing technique, you should at least find an entry for LoadLibrary and GetProcAddress. This will confirm that your dump is properly aligned on disk. (Additionally it should run if you execute it).
You can now import this in IDA and start the analysis.
API Hashing
For this part, you want to reverse the hashing algorithm.
First, find the function in charge of doing so. It should be easy to spot, as it should take a DWORD as argument, and a string (or another DWORD if the DLL name is hashed too). The string being the name of the DLL where the API comes from, and the DWORD being the API hash. Small tip: you can also list the functions by their number of call. The API resolving function should be called from a huge number of different place in the code, and is usually the one with the more references.
It is not impossible that the DLL name is decoded/decrypted just before the API resolving function.
If you are not sure, open it in a debugger, and execute random instructions just after the entrypoint, until you can observe the resolving mechanism.
If the DLL name is encoded/encrypted (witch is usually the case), start by reversing the algorithm. It can also be another hashing algorithm (or even the same one that the API). generally speaking, this part is easier to deal with, and you'll face some basic strings manipulations used to recover the DLL name.
Once you understand it, write a small function that is supposed to take an encrypted/encoded DLL name, and returns the plaintext one. You can then convert it to an IDAPython script that is going to label your idb with the plaintext names.
Half of the job is done here. Now, focus on the API hashing technique. Start by checking if the hash is not a standard one (CRC or others).
If this is done by hand, try to see if the hashing algorithm is not vulnerable to some reverse-manipulation that could lead to the recovery of the plaintext related hash.
If the algorithm is strong enough, you will not be able to recover the API name from the hash, but you'll have to bruteforce it. Luckily for you, you already have the name of the DLL where the API is going to be import. So you know that this hash resolve to something in a pre-defined list of API.
The idea is to re-implement the hashing algorithm, take the list of API defined in the target DLL, and pass them to your algorithm. Then, match the hash with the correct API name.
Once again, you can automate this with an IDAPython script to label your idb and makes things easier for you.
You now have something a bit more readable, and you should be able to understand the general goal of this malware.
RC4 encryption
This part is standard as the RC4 algorithm is well documented, not that hard to reverse, easy to spot, and very popular among malwares.
As always, try to identify your encryption function. You can search for some specific part of the algorithm (KSA, PRGA and the XOR) that have some hardcoded constants. The easy to stop one is the 'for' loop in the KSA, that should give you the following pseudo-code:
for (int i = 0; i < 0x100; i++) {
S[i] = i;
[...]
}
Every-time you see this, you can be sure that this is RC4.
Now, trace back those function to the main encryption wrapper. If this is standard, you should see two buffer being passed to the function: one for the encrypted content, and one for the key.
Once again, write a small script to decode this (plenty of libraries for RC4), and you should be able to recover the plaintext strings.
Don't forget to test your script on several encrypted strings, as the key may be different for each one.
Once you have marked up your idb with the plaintext string, you should have everything to analyze this malware.
