If I were to disassemble a malware sample in IDA without being in a VM, is there any way I can infect my machine?
The recommended workflow is to make the initial database (idb) by running IDA within a VM. That way you are safe even in case of a possible zero day.
The idb file can then be taken out of the VM and analyzed elsewhere. The idb does not contain runnable machine code, so you are safe.
If you want to perform any dynamic analysis on the malware, you need to use a VM.