Is it a good way to create Virus signature in MD5 or it is better in SHA1 or SHA256 and why it is?
Not in the long run.
Hashing of certain parts of files is still used to push data updates quickly when a new malicious sample shows up in the wild, to get the clients protected as soon as possible.
But malware nowadays is very polymorphic so hashing is easy to defeat.
Detailed analysis of samples take a lot of time often weeks or months and after that more generic/heuristic detections will be made which can cover the samples detected by hashes previously, so the hash signatures are dropped from the database gradually.
No form of hashing, when used alone, is a good way of developing an AV solution. While hashes can be useful for finding instances of a known sample across a network, heuristic and behavioural AV systems are far more likely to protect the host from infection.
That is exactly how antivirus programs work. Well, part of it. Kindly check this answer.
The thing is, MD5 hashing, although fast, is broken and have already produced collisions (different inputs producing the same MD5 hash). SHA1 and SHA256 are better in the sense that it is so much harder (but possible, as stated in the previous answer) to find collisions, that is to find different files that produce the same SHA hash.
Lastly, hashing used for malware detection is a very bad approach since it is easily defeated (polymorphic malware, malware builders, etc). Although if you are just exploring things, then YES, that is a valid way of creating an antivirus program.
Well, do you want it to be effective? Forget the fancy methods with hash collisions of my virus to a legitimate exectuable, it is much easier to circumvent all hash based Antiviruses.
The PE file format on windows and the ELF file format on Linux are used for executables (like .exe, .dll, .so, ...). Both formats specify the end of the file - everything after that is not parsed. That means an attacker can append random data and the file still runs like it did before.
Appending random data however changes the hash. If the attacker sends a different file to every victim that means that you would need to get each and every single one of those file to flag it as malicious.
And with some botnets having a million infected PCs that means you database would need to grow to that size - at which point you probably should search for another method of identification.
MD5 is used but, it's not the main method.. the Antivirus use special Algorithms in real-time protection. the anti-virus Does not depend on one Method. But, in my opening the antivirus Engine should use hashing to complete The mission Faster. -Translated by Google-
Using hashes is not a good idea since today's malwares use crypters which will use a function called USG (unique-stub-generater). This makes every file different, this means your hash will only work on exactly that file! Also new malwares have selfmodifying code which will make it harder, because the malware crypts itself every startup (or timeframe).
One thing I didn't see mentioned here is we discussed matching a virus signature via some hash - you could also consider matching instead your executable target to a known good hash for that executable target - this method would then not be sensitive to pure polymorphic virus behaviours - but of course one could find a collision in hash such that it still has the correct hash - so then using a a more computationally complex hash could help