The malware can propagate from the VM to the host in several circumstances:
- If there's a bug in the VM software. That's not very common, but it's possible.
- If there's a bug in the host OS. A bug that specifically allows malware to break out of a VM is unlikely but again possible.
- If there's a bug in the processor. That's even more unlikely but still not mathematically impossible.
- If there's a way for the malware to exchange information with the host. This can be over the network; in this respect, there's no difference between using a VM and using a separate physical machine, so you need to firewall the VM appropriately (allow only the bare minimum, don't do anything that might allow the server to hijack the client such as SSH with X forwarding). Another vector is via the VM tools such as file and clipboard sharing: the software running in the VM might read and write shared files, access the host clipboard, etc. So when running malware in a VM you should disable all these convenience tools (disable them in the VM configuration, it's not enough to refrain from installing the guest software since the malware could come with its own).
For more on that topic, read https://security.stackexchange.com/questions/9011/does-a-virtual-machine-stop-malware-from-doing-harm and https://security.stackexchange.com/questions/12546/is-it-safe-to-install-malware-in-a-vm
All in all, most malware can be analyzed safely in a properly configured virtual machine. Disable all guest features and don't connect to the VM with any protocol that allows the server to influence the client. Firewall the VM as if it was hostile (it is); you should use a host-only network so that you can easily control what goes there.
There's another reason why you might prefer a physical machine to a virtual machine. There's quite a bit of malware that attempts to detect whether it's being analyzed, and doesn't behave maliciously when it is. Obvious, common things that malware tries to detect is signs that it's running under a debugger: check if it's being ptraced (or the Windows equivalent), check for sudden pauses in the execution, … And some malware looks for telltale signs of a VM, such as drivers for hardware that common VM software emulates, or CPU oddities that are inexistent or rare outside emulated CPUs. This isn't to say that you cannot analyze malware in a VM: sometimes you can, sometimes you can't, it depends on the malware. If you start in a VM and find nothing you need to be prepared to move to a physical machine.
Note that there are risks with a physical machine too: malware could attempt to plant itself into one of the many pieces of firmware on the motherboard and peripherals. Toolkits for this are beginning to emerge, such as Rakshasa and Mebromi. So if you analyze some sophisticated malware on a physical machine, don't trust this physical machine any more, ever.