What is an entropy graph

Question

I am new to reversing and I see a tool Detect It Easy and it has a feature called Entropy. I want to know what it is used for?

Answer 1

it has a feature called Entropy. I want to know what it is used for?

For our purposes, entropy can be though of as information density or as a measure of randomness in information, which is what makes it useful in the context of reverse engineering and binary analysis.

Compressed and encrypted data have higher entropy than e.g. code or text data. In fact, compressed and encrypted data have close to the maximum possible level of entropy, which can be used as a heuristic to identify it as such in order to differentiate it from non-compressed/non-encrypted data.

Example use cases in reverse engineering:

• Malware Analysis - If we have an executable which has a header that can be parsed successfully and the program loads and runs without error, but the overall entropy level of the file is very high and the code can't be analyzed statically because the data outside of the file header and program headers looks random (hence the high entropy), it probably means that the executable is in fact compressed on disk and is decompressed at runtime. Executable compression complicates analysis, so it is a relatively common feature of programs developed for criminal purposes. If we want to analyze the code, its decompressed form need to be recovered somehow.

• Firmware Analysis - In systems with relatively severe hardware constraints, such as embedded systems, firmware updates are often delivered in compressed form in order to save space. In order to analyse the firmware, it first needs to be determined whether it is encrypted or compressed. One way to determine this is through performing an entropy analysis of the file. If the entropy is very high, it is a good sign that the file is indeed compressed or encrypted. To proceed with analysis of the actual firmware, it must first be decompressed/decrypted. If we have a block of data with very high entropy (i.e. close to random), it makes no sense to try to treat it as code and disassemble it, because the results will be meaningless nonsense.

• File Type Identification - Some file types can be identified on the basis of their overall entropy. For example, we can usually differentiate between image files (png, jpeg, etc) and compiled binaries (ELF, PE) because image files consist of compressed data and therefore (generally) have much higher entropy than compiled binaries.

Besides "Detect It Easy", tools such as binwalk, ent and binvis.io can assist with calculating file entropy. You can also build your own tools that do this.

Answer 2

Entropy is interpreted as the Degree of Disorder or Randomness

a high entropy means a highly disordered set of data

a low entropy means an ordered set of data

to address the comments
order here does not mean 'a' following 'a' kind of order it is to be interpreted as random / non random state of certain data

aaaabbbbccccdddd or "abcdabcdabcdabcd" or "adbcadbcadbcadbc" is a repetitive string whose entropy will be greater than
aaaaaaaabbbbcccd or any shuffled representation of this string

in the first string and its shuffled clones all have 4 chars with equal probability 4/16 or 1/4 or 25%
but in the second string char 'a' (8/16 ) or half of the data set has the highest probability
while 'c' (1/16) has the least or a very minuscule probability

entropy is a thermodynamic concept that was introduced to digital science (information theory) as a means to calculate how random a set of data is

simply put the highest compressed data will have the highest entropy

where all the 255 possible bytes will have equal frequencies

ie if 0x00 was seen 10 times in a blob 0x10 or 0x80 or 0xff will all be seen 10 times in the same blob

that is the blob will be a repeated sequence comprising of all bytes between of 0x0..0xff

while a low entropy blob will have a repeated sequence comprising only of a certain byte like 0x00 0r 0x55 or two bytes 0x0d0a ox222e etc or any series one less than 255 possible byte sequences

taking an algo from here and modifying it a little

import math
from collections import Counter
base = {
    'shannon' : 2.,
    'natural' : math.exp(1),
    'hartley' : 10.,
    'somrand'   : 256.
    }
def eta(data, unit):
    if len(data) <= 1:
        return 0
    counts = Counter()
    for d in data:
        counts[d] += 1
        ent = 0
        probs = [float(c) / len(data) for c in counts.values()]
        for p in probs:
            if p > 0.:
                ent -= p * math.log(p, base[unit])
    return ent
hes = "abcde\x80\x90\xff\xfe\xde"
les = "aaaaa\x61\x61\x61\x61\x61"
print ("=======================================================================================================")
print (" type      ent for hes                 hes                      ent for les            les")
print ("=======================================================================================================")
for i in base:
    for j in range(1,4,1):
        print (i ,' ', eta( j*hes,i) , '\t', (hes*j + (30 -j *10) *" " ) , ' ' , eta (j*les , i) ,'\t',  ("%s" % les*j )) 

you can see 'abcde\x80.....' is high entropy while 'aaaaa\x61...' is low entropy

:\>python foo.py
=======================================================================================================
 type      ent for hes                 hes               ent for les            les
=======================================================================================================
shannon   3.321928094887362      abcdeÿþÞ                   0.0    aaaaaaaaaa
shannon   3.321928094887362      abcdeÿþÞabcdeÿþÞ           0.0    aaaaaaaaaaaaaaaaaaaa
shannon   3.321928094887362      abcdeÿþÞabcdeÿþÞabcdeÿþÞ   0.0    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
natural   2.3025850929940455     abcdeÿþÞ                   0.0    aaaaaaaaaa
natural   2.3025850929940455     abcdeÿþÞabcdeÿþÞ           0.0    aaaaaaaaaaaaaaaaaaaa
natural   2.3025850929940455     abcdeÿþÞabcdeÿþÞabcdeÿþÞ   0.0    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
hartley   0.9999999999999998     abcdeÿþÞ                   0.0    aaaaaaaaaa
hartley   0.9999999999999998     abcdeÿþÞabcdeÿþÞ           0.0    aaaaaaaaaaaaaaaaaaaa
hartley   0.9999999999999998     abcdeÿþÞabcdeÿþÞabcdeÿþÞ   0.0    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
somrand   0.4152410118609203     abcdeÿþÞ                   0.0    aaaaaaaaaa
somrand   0.4152410118609203     abcdeÿþÞabcdeÿþÞ           0.0    aaaaaaaaaaaaaaaaaaaa
somrand   0.4152410118609203     abcdeÿþÞabcdeÿþÞabcdeÿþÞ   0.0    aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Answer 3

Just to add (small) piece of information to @blabb and @Johann Aydinbas answers, here is the cite from Practical Malware Analysis book regarding your question:

Packed executables can also be detected via a technique known as entropy calculation. Entropy is a measure of the disorder in a system or program [...]

Compressed or encrypted data more closely resembles random data, and therefore has high entropy; executables that are not encrypted or compressed have lower entropy. Automated tools for detecting packed programs often use heuristics like entropy.

You can find additional information here, under Increased entropy header.

Answer 4

Shannon's entropy comes from information theory. It is the measure of degree of randomness of text. If a string has greater Shannon's entropy it means it's a strong password. Principally, Shannon entropy equation provides a way to predict the average minimum number of bits required to encode a string of symbols, based on the frequency of the symbols.

Note that the base represents the number of possible characters. Base 2 can be replaced by any base. As can be seen in this code where it's replaced by 255.

This link has a simplest implementation of the algorithm for calculating entropy of novels and religious books. It tells us a lot. For example, that all the human generated books have nearly identical degree of fluctuation between disorder. It is a good feature of data. This is the link to code mentioned above. Information Entropy of different Books

Answer 5

First, you have to know that the term entropy is used to refer to two different concepts which are somehow related if you think twice, but as it is really not obvious at first sight, you should prefer to consider these two as different concepts.

Defining Entropy ?

The entropy that you want to know about can be defined as the amount of order, disorder, or chaos in a thermodynamic system.

On the other hand, the other entropy is coming from information theory and can be seen as a measure of the amount of information that can be stored in a system.

Why is it useful in RE ?

An entropy graph (to evaluate the amount of disorder) can be useful to detect the parts of the file that get close to random data. It will allow to detect the parts that have been encrypted/compressed and the parts that appear to be left untouched.

Indeed, a high disorder in data is exactly what you want to achieve when encrypting data. And, I told you that the two entropy definitions were related, if you store a lot of information in a minimum of bytes, it appears to be with a high level of disorder, so is compression...

That is why we use entropy graphs of files, be able to distinguish raw parts from encrypted/compressed sub-parts without any prior information of the file format.

An Example

For example, here is an entropy graph from the tool binwalk coming from another question from here:

Directly from this graph we can see that there is a first part that appear to be raw (probably asm opcodes if we look at the shape of the curve), then a part which is much likely encrypted (compression does not reach an entropy of 1 with such regularity usually), and finally padding with always the same byte (e.g. 0x00 or 0xff).