How do AV vendors create signatures for polymorphic viruses?

Question

I was working on a hobby AV project using ClamAV's engine. While ClamAV is a good open source engine, it has poor support for detecting polymorphic viruses. The latest updated version failed to detect many instances of Virut and Sality. How do commercial AVs detect polymorphic viruses?

Answer 1

To detect the polymorphic engine itself - properly - requires a copy of the engine. That was the case in the past, since the virus carried the engine in order to produce new copies of itself. The obvious attack against that is server-side polymorphism, where we ("we"=the AV industry) are left to guess at the capabilities of the engine, and which can change at any time, in response to our detections. However, back to the actual question: given an engine that can produce a sequence like this:

mov reg1, offset_of_crypted

[optional garbage from set 1]
[optional garbage from set 2]
[optional garbage from set 3]

mov reg2, key_for_crypted

[optional garbage from set 1]
[optional garbage from set 2]
[optional garbage from set 3]

mov reg3, size_of_crypted

[optional garbage from set 1]
[optional garbage from set 2]
[optional garbage from set 3]

[decrypt]

[optional garbage from set 1]
[optional garbage from set 2]
[optional garbage from set 3]

[adjust reg3]

[optional garbage from set 1]
[optional garbage from set 2]
[optional garbage from set 3]

[branch to decrypt until reg3 completes]

then we can analyse the opcodes that can produce the register assignments, and we know the set of garbage instructions, the decryption methods, the register adjustment, etc.

From there, we can use a state machine to watch for the real instructions, and ignore the fake ones. The implementation details of that are long and boring, and not suitable as an answer here. It's a one-engine one-algorithm relationship, in most cases.

As a result, the emulator became the most useful tool that we have against that attack, allowing us to essentially "let the virus decrypt itself" and then we can see what's underneath (the attack against that is obviously metamorphism, and the simplest implementation is at the source level rather than post-compilation).

So, in short, the answer is generally "we don't anymore". That is, we tend to no longer detect the polymorphic engine itself. There are of course exceptions to that, but they are few and far between these days.

Answer 2

Short answer: AV scanners does not use signatures for polymorphic samples. They use generic detection code.

Long answer: Polymorphic malware makes the code look different for different generations. Talking about file infectors (Sality and Virut), a generation is considered when a new file is infected. If sample A infects B, C and D, then this is the 1st generation. The code in this generation will, more or less, look similar. When samples B, C and D infects new files, E, F and G, then, this will be the 2nd generation, and the code will greatly differ between the 1st and the 2nd generation (depending on the file infector's quality). The same applies to later generations: new generations will be different to previous ones (again, it depends on the quality of the file infector, but is typically true).

So, if you decide to use a signature based on a stream of bytes or a stream of mnemonics you will only detect one generation, not the malware itself. Except for not so good polymorphic engines.

AV engineers typically write detection code (not signatures) and find evidences of the malware without relaying on the polymorphic code but, rather, on the specific code that doesn't change and/or in the semantics: although the code might appear to be very different it's always doing the same.

The evidences used can be, for example, the following ones:

• If the entry point is in the last section (like in the case of Sality/Virut).
• Specific instructions with the same semantics but using different registers (pushing in the stack the original entry point address if the file infector uses EPO).
• Specific values/offsets that are equal between different generations.

Another approach, not widely used because it's slow and is not enough but itself:

• Perform code analysis of the 1st functions and compare the control flow graphs. You can consider it a graph based signature.

Another approach widely used:

• Using an emulator (one of the big features ClamAV lacks) try to emulate a number of instructions of the malware and, then, find a specific buffer (stream of bytes) or a specific set of instructions. After that number of instructions, hopefully, the malware is already 'unpacked' in memory. If it was correctly emulated, of course.

So, if you want to write detection code for ClamAV for Sality or Virut I recommend you to use what they call bytecoded "signatures".

Answer 3

There are many kinds of polymorphic viruses, but generally most common solutions actually try to work around the problem and avoid detecting unknown samples on the users' machines. It's considered hard to detect viruses in real time on a live machine with little available resources without actually exposing the user to the malicious properties of the virus. Instead AVs prefer doing most of the heavy lifting in their comfort zones: internal labs and sandboxes.

Usually there a few directions to take:

1. Try to generate a signature that remains valid for as many samples as possible. I.e. signing bytes that aren't polymorphic or have only few variants. You'll need a big bulk of similar variants. AVs usually have algorithms to cluster and automatically generate signatures this way.
2. Try to remove the polymorphic layer and detect the underlying sample. UPX is a simple example since its very easy to unpack statically, so are some XOR encryption schemes.
3. Detect samples by means of dynamic analysis such as malicious activities/APIs, process injections and the likes. This comes with a lot of false positives, a known problem for HIPS systems.
4. Have your AV product upload unknown suspicious files to the backend, where samples are being analyzed by proprietary static and dynamic analysis machines, clustering algorithms and manual RE if needed. And then obviously signed the old way. KAV loves doing that to unknown files.

Many of these e usually combined, the 3rd method is used to detect suspicious files for the 4th. 1st and 4th usually have similar engines and flows, that start with static analysis because it's faster. Because most of the hard work is done in the AV vendor's labs these are a huge bottlenecks so speedups and prioritizing are big parts of the game