Where can I find benign samples with a high potential to false positive?

Question

In the pursuit and development of malware detection algorithms, often a big sample set of both malicious and benign samples is required. Both machine learning or similar automated techniques, as well as manual or partially manual signature generation, often require a good and varied example set of benign samples that are commonly mistaken as malicious.

Those samples are usually being automatically analyzed and then provided to a Reverse Engineer for further scrutiny, analysis and improvement of said malware detection algorithm.

Although finding malicious samples is frequently discussed (see multiple questions), discussion about benign sample sources seems lacking.

What are good benign sample repositories/feeds, preferably focused on potential/frequent false positive samples? Other sources or "retrival methods" (scraping) are also welcome!

Answer 1

You can find some of the samples from our work on deobfuscation at hxxp://www.cs.arizona.edu/projects/lynx-project/Samples/ (s/x/t/). The programs---a mix of Linux and Windows binaries---are all benign (you'll have to take my word for this :-), but use a variety of techniques commonly used by malware. These samples formed the basis for the results reported in the following papers:

• Yadegari et al, "A Generic Approach to Automatic Deobfuscation of Executable Code", Proc. IEEE Symposium on Security and Privacy, May 2015. These consist of programs obfuscated using tools such as VMProtect, ExeCryptor, and Themida; and several that use return-oriented programming.
• Qiu et al, "Identifying and Understanding Self-Checksumming Defenses in Software", Proc. Fifth ACM Conference on Data and Application Security and Privacy (CODASPY), March 2015. This is a collection of programs that use a variety of techniques, including self-modifying code, to implement self-checksumming-based anti-tampering.

We used to have a link to these samples on the project web page, but Google decided that we were distributing malware and put a big red warning next to their link (talk about false positives!), so we ended up removing the link. This is also the reason the link above has been disguised with "hxxp".

Answer 2

Malicious samples

My favorite source for malicious samples is a database i assembled myself by regulary pulling the free examples provided by VirusSign. Please not these samples do require registration, but come with reports indicating the type.

There may be no better archive for malicious samples than VirusTotal. If you don't happen to gain (intelligence) access there, you can still try open alternatives like Open Malware.

Generally, a good list for places to find malicious samples is here: LENNY ZELTSER

Benign Samples

As for benign samples, I propose extracting benign executables from a fresh OS installations. You can simply install the target in a virtual machine and get a script to extract them.

Another source is crawling sites like PortableFreeware. Please make sure you check if executables collected this way to work in your environment.

Possible false positives

Most malware archives label their samples based on the detection ratio and labels provided by different ant-malware products (like VirusTotal). Asssuming samples with very low detection ratio and generic labels to be false positives is probably the best way to go here. Also, data collected from HoneyPots may yield relatively higher false positive rates.

I often used the executables included in the Sysinternals Suite as a kind of 'false positives'. They are benign binaries utilizing system-near features in order to expose system internal data.

On a sitenode, please take time evaluating your dataset and finding biases hidden within. A lot of current research suffers from biased sample sets and unreproducible results.

Answer 3

Some of the techniques i've also came up with, for the sake of completeness (I won't accept my own answer):

download from github

Some Github repositories have multiple executables either as needed utilities or as build output. Scanning Github for those using a GithubDownloader proved being slow but effective, and I results are really likely to be benign.

collect all executables/hashes from accessible machines

Since I'm doing this as part of my job, I'm able to ask the IT department to collect hashes and samples from the multiple machines we have at my office. This was a good way to collect many executables for multiple versions of OSes with ease. Collecting hashes and downloading the ones available from VT was also a possibility (perhaps after further scrutiny).