Reverse engineering Windows Defender's signature for Metasploit Framework's metsrv.dll

Question

As a pentester for a consulting agency, it is part of our job to "evade" antivirus after gaining code execution on information systems. It is indeed necessary to prove exploitation of vulnerabilities, as opposed to simply reporting them. If an AV detects the tools we use, clients often will disregard the vulnerability because they are not convinced by the risk it causes.

We noticed that Windows Defender detects the metsrv.dll from Metasploit Framework in memory and kills our shell. The detection is done by mpengine.dll and is either done by some kind of criteria on the emulated binary or is a pattern of bytes in the DLL.

Now that the context of the question is clear, here is the actual question: how could I proceed to pin point exactly what the signature for this file is ?

Before your answer, here are the conclusions I have already come to:

• Windows Defender's scanning engine detects metsrv.dll from Metasploit at the moment it is loaded into memory.
• I can use Taviso's loadlibrary to reproduce the detection statically on GNU/Linux, using the following command:

./mpclient metsrv.x64.dll main(): Scanning metsrv.x64.dll... EngineScanCallback(): Scanning input EngineScanCallback(): Threat HackTool:Win64/Meterpreter.A!dll identified.

• I have reduced the sample size down from 200 kb to 13 kb using the GNU split utility and a "binary search" approach: the sample is split in two parts, both are fed to Windows Defender and the part that is detected is then split in two more parts. Repeat until it is possible, so as to minimize the test case.
• Disassembling mpengine.dll is not very helpful, because there are more than 30k functions found by IDA Pro in it.
• Code coverage analysis with Pin allows to reduce this set to 3k functions, which is still too much to analyze statically.
• mpengine.dll can be debugged in gdb. I put a watch point on the string "Win64/Meterpreter.A!dll" to see if I could find an interesting function that would read at this location, maybe at a time close to the verdict's time. Still lost because of the size of the code, even though the watch point is triggered two times.
• A script on GitHub called avwhy.py allows to infer signatures from AVs by changing one byte at a time and memorizing the ones that impact the AV's verdict. After running more than 16 hours, the tool returned me the whole file as part of the signature, which looks like a wrong result: it is improbable that using the split utility I have found the exact signature, because I expect to either have too much bytes or having removed useful bytes from the signature.

As you can see, I have spent numeral hours on this. The goal is to find the exact signature, not to evade it by applying some kind of transformation on metsrv.dll. I think it is a fun reverse engineering challenge but I am stuck for now.

What are the steps that I need to take in order to accomplish my goal?

Edit: In order to clarify what I'm trying to do, here is a self-published paper from Tavis Ormandy: Sophail: A Critical Analysis of Sophos Antivirus

At page 3, he shows the signature for the file "Attention 629". I am trying to achieve the same result. Of course I can attack the 3k functions and work from here, but I suppose Tavis had a more intelligent approach, and that is the type of answers I'm looking for.

Answer 1

One approach you might consider involves compiling different versions of metsrv.dll from source and then observing which ones are detected and which ones are not. For instance:

1. Comment out half the code in main and check if the resulting DLL is detected. As your goal is to evade static detection, it does not matter for now that the DLL is not fully functional.
2. If undetected, uncomment the first half of the code that was commented the first time and recheck the detection. Otherwise, comment out the second half of the code used in the previous step and recheck the detection.
3. Repeat this binary search approach until you have narrowed it down to a particular function that causes the detection.
4. Repeat steps 1-3 for code for that function and any further nested function calls. Eventually you should reach a point where there are no further function calls and where commenting out a single line of code gives changes the detection status of the file.
5. Check the pattern of bytes (machine code) corresponding to this last line of code and experiment with mutating it to test the robustness of mpengine's detection criteria.

You might also consider different compiler or build options (e.g. compile with debug symbols or different optimization level) with the original code to see if that makes the detection go away. If so, comparing the two binaries with fc or some other tool might point to the cause of detection.

Answer 2

I found all the signatures by performing an Out-of-Band attack on the engine and with the help of a huge set of mutated files. Lots of reverse-engineering as well. I had to triage a lot of false positives and it required manual work, so this is still not the ideal approach, but clearly more scalable than the other suggested ones.

I have shared some of the findings with the rapid7 team as you can see here: https://github.com/rapid7/metasploit-framework/issues/10815

Below is copy-paste for compliance with this forum's rules:

Let's say you have a FUD stager and that you encode the STAGES. Windows Defender can still detect your meterpreter session because it has a kernel-land callback to detect when images are loaded into memory (like every competitive AV). This is done with PsSetLoadImageNotifyRoutine &co.

Upon these events, Windows Defender performs a scan on the memory region where the image is loaded and search for STATIC signatures (far from machine learning and behavioural analysis...)

Common detected artifacts are:

• Constants such as strings ("[*] Attempting to add user %s to group %s on domain controller %s") => such a high score that it is almost an EICAR test, although WD checks that it belongs in a file that follows the PE format.
• Large integer (for instance 0x6A4ABC5B in ReflectiveLoader.h, which is a rot13 hash used to locate APIs that everyone copy pasted around for years);
• Pieces of hard-coded shellcode (each one in base_inject.c, for instance x48\xC1\xE8\x20);
• DLL exports (for instance the export "ReflectiveLoader" is searched both by WD and Kaspersky).

Some strings have a very low score but still matter in the verdict. If you have any of the elements above you will get flagged for sure, and removing the low scores ones won't have any impact. An example of such string is "scheduler_insert_waitable".

All in all, "hide yo wife hide your strings"...