As a pentester for a consulting agency, it is part of our job to "evade" antivirus after gaining code execution on information systems. It is indeed necessary to prove exploitation of vulnerabilities, as opposed to simply reporting them. If an AV detects the tools we use, clients often will disregard the vulnerability because they are not convinced by the risk it causes.

We noticed that Windows Defender detects the metsrv.dll from Metasploit Framework in memory and kills our shell. The detection is done by mpengine.dll and is either done by some kind of criteria on the emulated binary or is a pattern of bytes in the DLL.

Now that the context of the question is clear, here is the actual question: how could I proceed to pin point exactly what the signature for this file is ?

Before your answer, here are the conclusions I have already come to:

  • Windows Defender's scanning engine detects metsrv.dll from Metasploit at the moment it is loaded into memory.
  • I can use Taviso's loadlibrary to reproduce the detection statically on GNU/Linux, using the following command:

./mpclient metsrv.x64.dll main(): Scanning metsrv.x64.dll... EngineScanCallback(): Scanning input EngineScanCallback(): Threat HackTool:Win64/Meterpreter.A!dll identified.

  • I have reduced the sample size down from 200 kb to 13 kb using the GNU split utility and a "binary search" approach: the sample is split in two parts, both are fed to Windows Defender and the part that is detected is then split in two more parts. Repeat until it is possible, so as to minimize the test case.
  • Disassembling mpengine.dll is not very helpful, because there are more than 30k functions found by IDA Pro in it.
  • Code coverage analysis with Pin allows to reduce this set to 3k functions, which is still too much to analyze statically.
  • mpengine.dll can be debugged in gdb. I put a watch point on the string "Win64/Meterpreter.A!dll" to see if I could find an interesting function that would read at this location, maybe at a time close to the verdict's time. Still lost because of the size of the code, even though the watch point is triggered two times.
  • A script on GitHub called avwhy.py allows to infer signatures from AVs by changing one byte at a time and memorizing the ones that impact the AV's verdict. After running more than 16 hours, the tool returned me the whole file as part of the signature, which looks like a wrong result: it is improbable that using the split utility I have found the exact signature, because I expect to either have too much bytes or having removed useful bytes from the signature.

As you can see, I have spent numeral hours on this. The goal is to find the exact signature, not to evade it by applying some kind of transformation on metsrv.dll. I think it is a fun reverse engineering challenge but I am stuck for now.

What are the steps that I need to take in order to accomplish my goal?

Edit: In order to clarify what I'm trying to do, here is a self-published paper from Tavis Ormandy: Sophail: A Critical Analysis of Sophos Antivirus

At page 3, he shows the signature for the file "Attention 629". I am trying to achieve the same result. Of course I can attack the 3k functions and work from here, but I suppose Tavis had a more intelligent approach, and that is the type of answers I'm looking for.

  • Wlcome to SE RE. what part of the binary are those 13K? code? data? header data?
    – user22970
    Commented Oct 24, 2018 at 20:28
  • You're right, I forgot to mention that those 13K are the PE header and a part of the .text section.
    – plowsec
    Commented Oct 24, 2018 at 20:43
  • 1
    Have you looked at Alexei Bulazel's Windows Defender research? mpengine.dll has already been analyzed to some extent. You don't have to start from scratch. His tools are here: github.com/0xAlexei/WindowsDefenderTools
    – julian
    Commented Oct 24, 2018 at 21:45
  • Yes I have. However Alexei only released a small part of his research, like most BlackHat talkers do. If you dig his repo, you will find that he kept the essential code and tooling as "exercices for the reader". Furthermore, he benefited from mpengine.dll's debugging symbols, which I couldn't get my hands on. Also, I don't want to evade the emulator, but to find the static signature / detection pattern.
    – plowsec
    Commented Oct 24, 2018 at 22:02
  • 2
    I think you’re asking a wrong question. Sure, you can spend half a year reversing the detection like Alex did. But then you’ll be back to the original problem: how to evade the detection. And in fact you can probably achieve the second without knowing the exact answer to the first (even though it could give you some hints), so I personally would start on the evasion directly.
    – Igor Skochinsky
    Commented Oct 25, 2018 at 6:36

2 Answers 2


One approach you might consider involves compiling different versions of metsrv.dll from source and then observing which ones are detected and which ones are not. For instance:

  1. Comment out half the code in main and check if the resulting DLL is detected. As your goal is to evade static detection, it does not matter for now that the DLL is not fully functional.
  2. If undetected, uncomment the first half of the code that was commented the first time and recheck the detection. Otherwise, comment out the second half of the code used in the previous step and recheck the detection.
  3. Repeat this binary search approach until you have narrowed it down to a particular function that causes the detection.
  4. Repeat steps 1-3 for code for that function and any further nested function calls. Eventually you should reach a point where there are no further function calls and where commenting out a single line of code gives changes the detection status of the file.
  5. Check the pattern of bytes (machine code) corresponding to this last line of code and experiment with mutating it to test the robustness of mpengine's detection criteria.

You might also consider different compiler or build options (e.g. compile with debug symbols or different optimization level) with the original code to see if that makes the detection go away. If so, comparing the two binaries with fc or some other tool might point to the cause of detection.

  • 2
    @UnitedCoconut Your objections to this answer are made on irrelevant grounds. Scalability and access to source code are clearly not an issue in this case because we are dealing with a single open-source binary. Furthermore, your concerns here were not mentioned in your question. Please refrain from being rude. Regardless, I am not sure what you are expecting. There are no magic tricks or secret techniques here - there is no way around reversing mpengine.dll.
    – julian
    Commented Oct 25, 2018 at 23:25
  • 1
    @United Coconut If you're not even willing to fiddle around with a few lines of source code to find an answer when source is available, what hope do you have of finding an answer when it isn't?
    – Samantha
    Commented Oct 26, 2018 at 3:03
  • @SYS_V Apologies if I sounded rude, rest assured it wasn't my intention. Scability is an issue because I would prefer having lasting results due to the amount of time required to get these results, and my insctinct tells me mpengine.dll found several signatures in metsrv.dll. Samantha: I am willing to fiddle around with the code, but in a way that renders my work reusable and long-term resilient. I will clarify my answer based on your remarks.
    – plowsec
    Commented Oct 26, 2018 at 7:03

I found all the signatures by performing an Out-of-Band attack on the engine and with the help of a huge set of mutated files. Lots of reverse-engineering as well. I had to triage a lot of false positives and it required manual work, so this is still not the ideal approach, but clearly more scalable than the other suggested ones.

I have shared some of the findings with the rapid7 team as you can see here: https://github.com/rapid7/metasploit-framework/issues/10815

Below is copy-paste for compliance with this forum's rules:

Let's say you have a FUD stager and that you encode the STAGES. Windows Defender can still detect your meterpreter session because it has a kernel-land callback to detect when images are loaded into memory (like every competitive AV). This is done with PsSetLoadImageNotifyRoutine &co.

Upon these events, Windows Defender performs a scan on the memory region where the image is loaded and search for STATIC signatures (far from machine learning and behavioural analysis...)

Common detected artifacts are:

  • Constants such as strings ("[*] Attempting to add user %s to group %s on domain controller %s") => such a high score that it is almost an EICAR test, although WD checks that it belongs in a file that follows the PE format.
  • Large integer (for instance 0x6A4ABC5B in ReflectiveLoader.h, which is a rot13 hash used to locate APIs that everyone copy pasted around for years);
  • Pieces of hard-coded shellcode (each one in base_inject.c, for instance x48\xC1\xE8\x20);
  • DLL exports (for instance the export "ReflectiveLoader" is searched both by WD and Kaspersky).

Some strings have a very low score but still matter in the verdict. If you have any of the elements above you will get flagged for sure, and removing the low scores ones won't have any impact. An example of such string is "scheduler_insert_waitable".

All in all, "hide yo wife hide your strings"...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.