The title mentions "sandbox" but VMWare or QEMU is usually not called that, so the question seems to be more about "how do I analyze it without a danger of infection?"
There are three broad categories of tools and approaches you could take here.
1. User-mode sandboxes
A user-mode sandbox basically runs the sample being investigated but intercepts all or at least the most critical APIs that access the host OS, neutralizes them and modifies the results to fool the software into thinking it's running all alone. One of the most popular such tools seems to be Sandboxie, but there exist others such as PyBox. Usually can be detected by the malware pretty easily and there's always a danger that an unemulated API will let the code to run amok.
2. Virtual machines and emulators
These go a bit deeper and try to emulate not only APIs but execution of the actual code. As well, usually you need to run a full OS inside the emulator and can't use the host OS as is (could be an advantage or disadvantage depending on your goals). These can be further subdivided in three categories by approach used for emulation:
These include VMWare, VirtualBox and VirtualPC. They use virtualization features of the recent processors to run most of the code natively and only emulate memory or hardware accesses. This makes them fast but in theory can lead to code escaping the VM in case of implementation bugs.
b) dynamic translation
This approach is used by QEMU. It translates each basic block into a sequence of native CPU's instructions and executes that. This approach allows it to reasonably fast emulate many different architectures, however the timing may differ quite a lot from the original. I don't think I've heard of any VM escaping bugs but it's possible to detect it.
c) full emulation
This is used by Bochs. It fully emulates each separate instruction one by one, as they're being executed. This makes it somewhat slower than other solutions but allows it to achieve almost perfect emulation of even the most low-level details. It's also probably the safest regarding to VM escaping bugs. There were some implementation bugs that could be used to detect it but I think most of them have been fixed.
3. Static analysis
The best way to avoid the break out is to not run the code at all! Also, static analysis allows you to look at the complete code of the binary and see all of it, even the code paths which are not taken by running it and you sidestep all runtime checks and detections.
Unfortunately, static analysis can be hampered by packing or obfuscation used in the malware. So it's often necessary to combine several approaches. I'm not a professional analyst, but I do dabble in some malware analysis. My workflow usually goes like this:
- Open sample in IDA.
- if it looks packed or encrypted, use Bochs debugger to emulate its execution and let it unpack itself.
- When it's finished (usually it's obvious), I take a memory snapshot, stop debugging, and continue analysis statically.
- If the unpacked code contains another embedded file (pretty common situation), saved it into a separate file and go to step 1