I'd recommend using something such as OllyDbg or IDA PRO rather than just plain HexEditor since just using HexEditor you've got no idea what section of the PE you are editing. For example, you could be editing the
.text PE section and editing an essential part of the program rather than
.idata or .data section which just store variable data.
So, few methods an AV could use for detection of malware are:
- Checksum of the whole file for an well-known piece of malware
- Checksum the
.text (Can also be named
.code) section of the PE as these pages are not flagged with WRITE permissions by default
- Checksum of the pattern that's malicious for example
WriteProcessMemory are often used for injecting DLLs or code into other processes. We could just create signature for that part of the code
Bypassing Method 1:
By randomly HexEditting you would manage to avoid AV detection method 1 (see above) but method 2 and 3 would detect the malware still. If you didn't overwrite anything essential for your program to execute.
Bypassing Method 2:
Rewrite a simple instruction or make a
JMP to code-cave and put overwritten code there and JMP back to next instruction. So, you would be executing exact same code but different flow and .text section checksum would be different.
Bypassing Method 3:
To be able to bypass method 3 - You would need to add junk code between each call or add a JMP to a code cave and have the instruction set there. So, the code would still be the same but byte signature would be completely different.
Your best to learn about assembly language and learn how to use IDA PRO or OllyDbg and learn about patching programs before you move onto something more complex like bypassing AV.