0

How should I decide file is malware or not. Before reverse engineering any file, I should be suspect the file. What are the various ways to decide if exe is malicious?

Thanks

closed as too broad by Jason Geffner, Peter Andersson Jun 5 '14 at 14:08

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

2

This is a million dollar question and I doubt anybody will be able to provide a convincing answer. There are many methods used by antivirus software & analysts.

One is to perform a stupid matching with known malware signatures. Another could be to statically analyze the application, extract the control flow (by reconstructing the CFG) of the application and deploy heuristics & pattern matching algorithms in order to determine if the application performs suspicious tasks. This can usually be done after analyzing known malware & building profiles of their behaviors (suspicious syscall call graph, ...).

There are numerous techniques and this field is still open for research (this document describes an interesting one), some are extremely advanced and may necessitate sharp mathematical skills, and others are hard to program. Mainly, because none of them follow the standard algorithmic approach and rather use machine learning, genetic algorithms, and sometimes AI.

I suggest you going through this Securelist article, and this publication too if you're interested by more material.

Not the answer you're looking for? Browse other questions tagged or ask your own question.