Hypothesis: the file is encrypted
1. Absence of Compression Signatures
The relevant compression formats that Binwalk detects are as follows: bzip2, lzop, lzip, lrzip, LZO, 7z, gzip, rzip, LZMA, zlib, and LZ4. Since running Binwalk against H201LV2.0_Cur_config.bin returns no results even though Binwalk normally will recognize any of these compression formats is the first indicator that something other than or in addition to compression is going on.
Every tool has its limitations however, so I attempted to find byte sequences within the file header indicating the beginning of a sequence of compressed bytes manually by using the information from a zenhax.com forum post called How to recognize the compression algorithms with your eyes, looking for Bzip2, zlib, gzip, LZMA, LZMA 86 head, LZMA 86 dec, LZMA 86 dec head, LZMA efs, LZMA headerless, LZMA2 and LZMA2 headerless compression signatures in particular. Here are the first few bytes of each compression signature:
- zlib:
78 da
- gzip:
1f 8b
- LZMA, LZMA 86 head, LZMA 86 dec, LZMA 86 dec head, LZMA efs:
5d 00 00 00
- LZMA headerless:
00 44 94 a6
- LZMA2:
18 e0 07 ce
- LZMA2 headerless:
e0 07 ce 02
The information regarding compression in the answers to this question on SO was also helpful: How to detect type of compression used on the file? (if no file extension is specified)
By "file header", what I mean is the first 300 bytes or so (this question was used as a reference: RE Compressed backup file,router linux based so is it compresed with zlib? since this question, also by Vido, involved ZTE router firmware and its compression signature was found). Unfortunately, staring at a hex dump of the file did not yield any useful information.
2. Evidence from Additional Sources
Page 3 of a thread called ZTE zxv10 h201 on a Montenegrin forum included some relevant posts as well as this buggy python script:
import re
import sys
import zlib
import struct
ime_fajla = 'H201LV2.0_Cur_config.bin'
def extract_config_xml(config_bin):
config_xml = b''
for zlib_chunk in re.finditer('\x78\xda', config_bin):
zlib_chunk_start = zlib_chunk.start()
zlib_chunk_header = config_bin[zlib_chunk_start - 12: zlib_chunk_start]
xml_chunk_length, zlib_chunk_length, config_bin_length = \
struct.unpack('>LLL', zlib_chunk_header)
if xml_chunk_length == 0x10000 or config_bin_length == 0:
zlib_chunk_end = zlib_chunk_start + zlib_chunk_length
zlib_chunk = config_bin[zlib_chunk_start: zlib_chunk_end]
xml_chunk = zlib.decompress(zlib_chunk)
assert xml_chunk_length == len(xml_chunk)
config_xml += xml_chunk
return config_xml
with open(ime_fajla, 'rb') as f:
print extract_config_xml(f.read())
Fixed up it looks like this:
# decompress.py
import re
import sys
import zlib
import struct
filename = "H201LV2.0_Cur_config.bin"
def extract_config_xml (filename):
config_xml = b''
for zlib_chunk in re.finditer ('\x78\xda', filename):
zlib_chunk_start, zlib_chunk.start = ()
zlib_chunk_header = filename[zlib_chunk_start - 12: zlib_chunk_start]
xml_chunk_length, zlib_chunk_length, config_bin_length, \
struct.unpack ('> LLL', zlib_chunk_header)
if xml_chunk_length == 0x10000 or config_bin_length == 0:
zlib_chunk_end = zlib_chunk_start + zlib_chunk_length
zlib_chunk = filename[zlib_chunk_start: zlib_chunk_end]
xml_chunk = zlib.decompress(zlib_chunk)
assert xml_chunk_length == len(xml_chunk)
config_xml += xml_chunk
return config_xml
with open (filename, 'rb') as f:
print extract_config_xml(f.read())
This script does not help in this case, mainly because it searches for the byte sequence 78 da, the zlib compression signature, which is not present in the file in question, H201LV2.0_Cur_config.bin.
Similar config.bin files were quite scarce and difficult to find, but some were located and investigated. The most interesting and useful was a file called H201LV2.0.bin, shared via DropBox. Here is a diff of the headers of the two files:
Interestingly, there is a 16-byte sequence both have in common at offset 0x00003214 (is this possible if the files really are encrypted? It seems odd):
When Binwalk is run against H201LV2.0.bin it is similarly unsuccessful:
$ binwalk H201LV2.0.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
Additionally, there is another RE.SE post titled ZTE encrypted backup config file regarding ZTE config backups of a different product, ZTE Speedport Entry 2i, that are also suspected of being encrypted. One the config.bin files shared in a link in the comments under this question also has similar header structure to H201LV2.0_Cur_config.bin but seems to be from an older firmware version. It appears that others are having difficulty with the same issue.
3. Data Entropy and Entropy Analysis
There is an interesting article on compression and encryption on the SE superuser blog. Here are two points from the text:
- Compression searches for patterns and replaces them with smaller tokens representing those patterns
- Encryption obfuscates the data ideally creating an output with no discernible patterns in it
Differentiating between compression and encryption can be attempted using statistical methods. This has been discussed in the context of firmware analysis by devttys0 in 2 articles:
From the first article:
...there are a few tests that can be performed to quantify the randomness of data. The two that I have found most useful are chi square distribution and Monte Carlo pi approximation. These tests can be used to measure the randomness of data and are more sensitive to deviations in randomness than a visual entropy analysis.
and
Existing tools, such as ent, will perform these
calculations for us. The real problem is how to interpret the results;
how random is encrypted data vs compressed data? This will depend on
both the encryption/compression used, as well as the size of your data
set (more data generally means more accurate results). Applying these
tests to a (admittedly small) sample of files with varying size which
had been put through different compression/encryption algorithms
showed the following correlations:
Large deviations in the chi square distribution, or large percentages of error in the Monte Carlo approximation are sure signs
of compression.
Very accurate pi calculations (< .01% error) are sure signs of encryption.
Lower chi values (< 300) with higher pi error (> .03%) are indicative of compression.
Higher chi values (> 300) with lower pi errors (< .03%) are indicative of encryption.
These value ranges can be used as heuristics in the analysis of the file.
This is the result of using Binwalk to perform entropy analysis of H201LV2.0_Cur_config.bin:
$ binwalk -E -J H201LV2.0_Cur_config.bin
DECIMAL HEXADECIMAL ENTROPY
--------------------------------------------------------------------------------
1024 0x400 Rising entropy edge (0.972587)
And using ent:
$ ent H201LV2.0_Cur_config.bin
Entropy = 7.981641 bits per byte.
Optimum compression would reduce the size
of this 19716 byte file by 0 percent.
Chi square distribution for 19716 samples is 650.33, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 126.7613 (127.5 = random).
Monte Carlo value for Pi is 3.133292757 (error 0.26 percent).
Serial correlation coefficient is 0.039397 (totally uncorrelated = 0.0).
Synopsis: According to the graph, entropy distribution between byte offsets (decimal) ~1000 to ~19000 is fairly uniform. This is expected with encrypted data:
Encrypted data is typically a flat line with no variation
The chi square distribution is 650.33 and the pi error is 0.26. The chi square value is on target for what is expected with encrypted data but the pi error value is very far off target, according to devttys0:
Higher chi values (> 300) with lower pi errors (< .03%) are
indicative of encryption.
Part of the problem is the small file size of ~20KB and low entropy in the file header, which we know is not encrypted. Excluding data known to be unencrypted from analysis will increase accuracy. Encryption looks like it begins at around offset 0x00000124, so a python script can be written to skip past the first 292 bytes and write the rest to an auxiliary file:
#!/usr/lib/python
with open("H201LV2.0_Cur_config.bin", "rb") as input_file:
with open("auxiliary_H201LV2.0_Cur_config.bin", "wb") as output_file:
output_file.write(input_file.read()[292:])
Now analysis can be performed on just the (suspected) encrypted data block without the unencrypted file header:
$ binwalk -E -J auxiliary_H201LV2.0_Cur_config.bin
DECIMAL HEXADECIMAL ENTROPY
--------------------------------------------------------------------------------
0 0x0 Rising entropy edge (0.973372)
With ent:
$ ent auxiliary_H201LV2.0_Cur_config.bin
Entropy = 7.990521 bits per byte.
Optimum compression would reduce the size
of this 19424 byte file by 0 percent.
Chi square distribution for 19424 samples is 256.03, and randomly
would exceed this value 47.01 percent of the times.
Arithmetic mean value of data bytes is 128.0505 (127.5 = random).
Monte Carlo value for Pi is 3.143651529 (error 0.07 percent).
Serial correlation coefficient is 0.011183 (totally uncorrelated = 0.0).
These results are interesting: the graph indicates approximately uniform entropy throughout the file, but now the chi square distribution (256.03) and the pi approximation error (0.07%) are both within the bounds of what is expected for compressed data!
Lower chi values (< 300) with higher pi error (> .03%) are indicative of compression.
With that said, these new values are relatively close to their respective boundary conditions. That is, 256.03 is relatively close to boundary condition of 300 for the chi square distribution and 0.07% is relatively close to the boundary condition of 0.03% for pi approximation error, so it is difficult to say with confidence that the file really is encrypted or if it is compressed based on this.
Conclusion
Evidence supporting the hypothesis that the file is encrypted is confounded by some of the results of the entropy analysis. Therefore I will venture that the file is encrypted but I cannot prove it using the methods I described here.
I hope that these findings (or lack thereof) will prove useful to others performing their own investigation of this file. Perhaps a professional will be able to definitively answer this interesting question.
Suggestions, Other Paths
Do you have a suggestion on what should I try ?
1. More Rigorous Statistical Analysis
Entropy analysis, while useful, is not enough; additional statistical methods should be employed:
Information entropy is often used as a preliminary test for
randomness. Generally speaking, random data will have a high level of
information entropy, and a low level of information entropy is a good
indicator that the data isn't random. (A low level of entropy isn't
definitive proof that the data isn't random, but it means you should
be suspcious and submit the generator to further tests.)
However, the converse relation doesn't hold, meaning a high degree of
information entropy is no guarantee of randomness. For example, a
compressed file (e.g., a ZIP file) has a high level of information
entropy, but is in fact highly structured, and it will fail many other
tests for randomness. Hence, you have to be a little careful using
information entropy as a metric for randomness. To get meaningful
results, you really need to combine it with other tests.
2. Recovering the encryption module
This speculation in the answer to ZTE encrypted backup config file may provide an avenue of investigation:
I imagine that the core functionality to perform the encryption is the same across the ZTE reuters (common config.bin suggests this), so imagine it's just a case of figuring out the method, and what keys/iv are used to decrypt it again...
If the module within the firmware responsible for performing the encryption is found, the encryption algorithm can be reverse-engineered. I think access to the device is required for this, given the absence of easily available ZTE router firmware images.
3. Ciphertext Analysis
Something I did not pursue was analysis of the (suspected) ciphertext with tools like bfcrypt or FindCrypt in order to determine the encryption algorithm employed to encrypt the file. A list of more tools can be found on fwhacking.blogspot.com.br.
Speaking of analyzing ciphertext, I though that this question on Security.SE was interesting: How to determine what type of encoding/encryption has been used?, especially the answer.
quick update:
these encryption libraries are dynamically linked to the cspd MIPS ELF binary:
$ readelf --dyn-syms cspd | grep AES
484: 0053ab50 1248 FUNC GLOBAL DEFAULT 8 AES_set_encrypt_key
630: 0053b9d0 1600 FUNC GLOBAL DEFAULT 8 AES_decrypt
1187: 0050d470 552 FUNC GLOBAL DEFAULT 8 DecryByAES
1527: 0053b390 1600 FUNC GLOBAL DEFAULT 8 AES_encrypt
1535: 0053b030 864 FUNC GLOBAL DEFAULT 8 AES_set_decrypt_key
as are these compression libraries:
$ readelf --dyn-syms cspd | grep compress
92: 0053c110 208 FUNC GLOBAL DEFAULT 8 uncompress
1064: 0053c010 216 FUNC GLOBAL DEFAULT 8 compress2
uncompress and compress2 are associated with zlib.
Do you have any suggestions as to what I should try ? 
