How does PeID calculate entropy?

Question

I'm curious about the entropy field value that shows up when a binary is dropped into PeID. Does someone know how it arrives upon this value?

Answer 1

This is exactly a problem I dealt with several years ago because we also wanted to implement a close-as-possible solution for a project formerly based on PeID.

Fortunately, in those days I found an old discussion in the PeID forum where someone asked the same question and got some hints from the author (not the exact code). I still have the link, but the entire site has ceased to exist :(

Those days I managed to re-implement a version of that algorithm which yielded the same values most of the time. The core points of original PeID's entropy implementation are:

• It indeed uses the (Order-0) Shannon formula (see Guntrams hint) as core algorithm
• Such an entropy is calculated for each PE section independently
• use the raw size of each PE section (i.e. size on disk) as starting point for the data amount, then subtract all 0-Bytes at the very end from this value before calculating the Shannon entropy.
• sections may be ignored in the further calculation if following conditions apply:
• • the section is the resource section
• • no (remaining) data in the section (after null byte subtraction)
• the overall entropy is built by a weighted combination of all single section entropy values based on their (effective) lengths.

This way you should be able to get very similar results. But expect deviations in non-standard samples (e.g. corruped/tampered PE files), i.e. you should expect and handle cases like implausible section sizes or files where all sections are excluded from the calculation...

I hope this helps a bit and you can now successfully try this yourself... ;)

Answer 2

You'd have to ask the creator of PeID which algorithm he uses.

The standard formula for this is the Shannon formula: H(x) = - sum i=1 to n p(i)*log2(p(i)).

Why can't i use LaTeX here?

An easy way to determine the entropy of a file is trying to compress it, using one of the standard compression Tools. Low entropies lead to better compression.

Especially when you try to determine which sections are code and which are data, the compression test is a better indicator. The original shannon formula does not take into account if there are combinations of multiple bytes that occur often (like the push/pop sequences of function starts/ends), but compression algorithms recognize those.

Answer 3

a generic entropy calculation looks like this

create a histogram of the contents of the binary max 256 different bytes (0x0..0xff)
calculate (log base 2 of probability of each byte multiplied by its count) add them all up and divide by filesize

byte   count freq?        log2 * count
0x00     100   (100/filesize)     log2(freq[0]?)*count   
0x01     045   (045/filesize)     log2(freq[1]?)*count    
................................................    
0xfe     020   (020/filesize)     log2(freq[0xfe]?)*count    
0xff     180   (180/filesize)     log2(freq[0xfF]?)*count 
entropy =            ^sum(0x0..0xff)^/filesize       

a sample code

#include <stdio.h>
#include <math.h>
#include <stdlib.h>
int main (int argc,char *argv[]) {
  if(argc !=2 ){ printf ("usage %s file",argv[0]);return 0;}
  FILE * fp = 0;errno_t err = 0; unsigned long fsize = 0; float ent =0;
  if (((err = fopen_s(&fp,argv[1],"rb")) == 0)  && (fp !=0)){
    fseek(fp,0,SEEK_END);
    fsize = ftell(fp);
    fseek(fp,0,SEEK_SET);
    unsigned char * buff = (unsigned char *)calloc(fsize,sizeof(char));
    if(buff){
      fread(buff,fsize,sizeof(char),fp);
      fclose(fp);
      unsigned long hist[256]= {0};
      for(unsigned int i =0; i < fsize; i++){
        hist[buff[i]]++;        
      }
      for(unsigned int i=0;i<256;i++){//log b2 (n) = loge(n)/loge(2)
        ent+=((log((float)hist[i]/fsize)/log((float)2))*hist[i]);
      }
      printf("%f\n",-ent/fsize);
      free(buff);
    }    
  }   
}

result

:\>ls -l *
-rw-rw-rw-  1 Admin 0 867 2015-06-30 03:30 entropy.cpp 

:\>..\compile.bat    
:\>if "C:\Program Files\Microsoft Visual Studio 10.0\VC\" == "" ()    
:\>cl /Zi /EHsc /nologo /W4 /analyze *.cpp /link /RELEASE
entropy.cpp  

:\>ls -l *
-rw-rw-rw-  1 Admin 0     867 2015-06-30 03:30 entropy.cpp
-rwxrwxrwx  1 Admin 0  135680 2015-06-30 03:32 entropy.exe
-rw-rw-rw-  1 Admin 0    5804 2015-06-30 03:32 entropy.obj
-rw-rw-rw-  1 Admin 0 1297408 2015-06-30 03:32 entropy.pdb
-rw-rw-rw-  1 Admin 0   53248 2015-06-30 03:32 vc100.pdb    

:\>entropy.exe entropy.exe
6.577609 

was tinkering with powershell and these two lines of script will calculate the same entropy as the c code above

PS > cat .\entropy.ps1
$s=($f=gc -Encoding Byte $args)|measure;$g=$f|group -AsHashTable
 0..255|%{ $e+=(((([math]::Log( $g[$_].count / $s.count)) /
                    [math]::Log(2.0)) * $g[$_].count)/$s.count)};-$e
PS > .\entropy.ps1 .\entropy.exe
6.57760952796101