First, let's see UPX structure.
UPX Structure
Prologue
CMP / JNZ for DLLs parameter checks
Pushad, set registers
optional NOP alignment
Decompression algorithm
Call/Jumps restoring
- UPX transform relative calls and jumps into absolute ones, to improve compression.
Imports
- load libraries, resolve APIs
Reset section flags
Epilogue
- clean stack
- jump to the original EntryPoint
For more details, here is a commented IDA (free version) IDB of a UPX-ed PE.
modified UPX variants
Simple parts like prologue/epilogue are easy to modify, and are consequently often modified:
- basic polymorphism: replacing an instruction with an equivalent
- moving them around with jumps
Complex parts like decompression, calls restoration, imports loading are usually kept unmodified, so usually, custom code is inserted between them:
- an anti-debug
- an extra xor loop (after decompression)
- a marker that will be checked further in the unpacked code, so that the file knows it was unpacked.
faking
As the prologue doesn't do much, it's also trivial to copy it to the EntryPoint of a non UPX-packed PE, to fool identifiers and fake UPX packing.