1

I'm curious about the entropy field value that shows up when a binary is dropped into PeID. Does someone know how it arrives upon this value?

4

This is exactly a problem I dealt with several years ago because we also wanted to implement a close-as-possible solution for a project formerly based on PeID.

Fortunately, in those days I found an old discussion in the PeID forum where someone asked the same question and got some hints from the author (not the exact code). I still have the link, but the entire site has ceased to exist :(

Those days I managed to re-implement a version of that algorithm which yielded the same values most of the time. The core points of original PeID's entropy implementation are:

  • It indeed uses the (Order-0) Shannon formula (see Guntrams hint) as core algorithm
  • Such an entropy is calculated for each PE section independently
  • use the raw size of each PE section (i.e. size on disk) as starting point for the data amount, then subtract all 0-Bytes at the very end from this value before calculating the Shannon entropy.
  • sections may be ignored in the further calculation if following conditions apply:
    • the section is the resource section
    • no (remaining) data in the section (after null byte subtraction)
  • the overall entropy is built by a weighted combination of all single section entropy values based on their (effective) lengths.

This way you should be able to get very similar results. But expect deviations in non-standard samples (e.g. corruped/tampered PE files), i.e. you should expect and handle cases like implausible section sizes or files where all sections are excluded from the calculation...

I hope this helps a bit and you can now successfully try this yourself... ;)

2
  • Did you check the Wayback Machine for that page?
    – Jongware
    Jun 29 '15 at 21:53
  • Yes, it's not indexed :(
    – Chris14
    Jun 29 '15 at 21:54
1

You'd have to ask the creator of PeID which algorithm he uses.

The standard formula for this is the Shannon formula: H(x) = - sum i=1 to n p(i)*log2(p(i)).

Why can't i use LaTeX here?

An easy way to determine the entropy of a file is trying to compress it, using one of the standard compression Tools. Low entropies lead to better compression.

Especially when you try to determine which sections are code and which are data, the compression test is a better indicator. The original shannon formula does not take into account if there are combinations of multiple bytes that occur often (like the push/pop sequences of function starts/ends), but compression algorithms recognize those.

1

a generic entropy calculation looks like this

create a histogram of the contents of the binary max 256 different bytes (0x0..0xff)
calculate (log base 2 of probability of each byte multiplied by its count) add them all up and divide by filesize

byte count freq? log2 * count

0x00 100 (100/filesize) log2(freq[0]?)*count 0x01 045 (045/filesize) log2(freq[1]?)*count ................................................ 0xfe 020 (020/filesize) log2(freq[0xfe]?)*count 0xff 180 (180/filesize) log2(freq[0xfF]?)*count

entropy = ^sum(0x0..0xff)^/filesize

a sample code

#include <stdio.h>
#include <math.h>
#include <stdlib.h>
int main (int argc,char *argv[]) {
  if(argc !=2 ){ printf ("usage %s file",argv[0]);return 0;}
  FILE * fp = 0;errno_t err = 0; unsigned long fsize = 0; float ent =0;
  if (((err = fopen_s(&fp,argv[1],"rb")) == 0)  && (fp !=0)){
    fseek(fp,0,SEEK_END);
    fsize = ftell(fp);
    fseek(fp,0,SEEK_SET);
    unsigned char * buff = (unsigned char *)calloc(fsize,sizeof(char));
    if(buff){
      fread(buff,fsize,sizeof(char),fp);
      fclose(fp);
      unsigned long hist[256]= {0};
      for(unsigned int i =0; i < fsize; i++){
        hist[buff[i]]++;        
      }
      for(unsigned int i=0;i<256;i++){//log b2 (n) = loge(n)/loge(2)
        ent+=((log((float)hist[i]/fsize)/log((float)2))*hist[i]);
      }
      printf("%f\n",-ent/fsize);
      free(buff);
    }    
  }   
}

result

:\>ls -l *
-rw-rw-rw-  1 Admin 0 867 2015-06-30 03:30 entropy.cpp 

:\>..\compile.bat    
:\>if "C:\Program Files\Microsoft Visual Studio 10.0\VC\" == "" ()    
:\>cl /Zi /EHsc /nologo /W4 /analyze *.cpp /link /RELEASE
entropy.cpp  

:\>ls -l *
-rw-rw-rw-  1 Admin 0     867 2015-06-30 03:30 entropy.cpp
-rwxrwxrwx  1 Admin 0  135680 2015-06-30 03:32 entropy.exe
-rw-rw-rw-  1 Admin 0    5804 2015-06-30 03:32 entropy.obj
-rw-rw-rw-  1 Admin 0 1297408 2015-06-30 03:32 entropy.pdb
-rw-rw-rw-  1 Admin 0   53248 2015-06-30 03:32 vc100.pdb    

:\>entropy.exe entropy.exe 6.577609

was tinkering with powershell and these two lines of script will calculate the same entropy as the c code above

PS > cat .\entropy.ps1
$s=($f=gc -Encoding Byte $args)|measure;$g=$f|group -AsHashTable
 0..255|%{ $e+=(((([math]::Log( $g[$_].count / $s.count)) /
                    [math]::Log(2.0)) * $g[$_].count)/$s.count)};-$e
PS > .\entropy.ps1 .\entropy.exe

6.57760952796101

1
  • There's a problem with such method. If you create a file that consists of 256 blocks, each one 256 bytes long and filled with the same byte value (256 zeros, 256 0x01s, .., 256 0xFFs), it will give the result of 8.0 - the highest entropy, however such a file is very well compressible (64K become 420 bytes with Winrar). This method doesn't count the distribution of bytes within a file.
    – mistika
    Sep 8 '17 at 17:09

This site is temporarily in read only mode and not accepting new answers.