Reverse Engineering Stack Exchange is a question and answer site for researchers and developers who explore the principles of a system through analysis of its structure, function, and operation. It only takes a minute to sign up.
Sign up to join this community
When reversing a router firmware based on DD-WRT I came across a mention of "webcomp". It seems it's used for storing internal HTML files for the web interface. What exactly it is and how can I extract those files from the firmware image?
NOTE: This is a self-asked and answered question to add to the knowledge based.
/*
* webcomp -- Compile web pages into C source
*
* Copyright (c) GoAhead Software Inc., 1995-2000. All Rights Reserved.
*/
Webcomp, or 'web compiler', is a very simple tool written by GoAhead Software during the period of 1995-2000. It 'compiles' web pages into a single binary blob, adding an index of files in the blob to the httpd for use at runtime.
Webcomp is available in the DD-WRT source code. It clearly was created in a 'just hack it out' style, without much concern for its appearance. It does the job though, which is what matters.
DD-WRT uses this tool to 'protect' their web management console. Since their web management console is derived from GPL'd open source, has had countless contributors over the years, and webcomp itself is open source, I do not believe there is any legal restriction against documenting the mechanism they use.
Binary Blob. Normally stored as a file at /etc/www, this is a simple concatenation of all web pages on the device. It is not compressed or obfuscated. Nor should it be. The file systems used by embedded linux firmwares, such as squashfs, feature compression as a primary attribute. Therefore, compressing this blob would be redundant and would harm the final compression ratio of the file system. Similarly, obfuscation of the block would substantially affect the compressibility. However, there is a commented out section in the webcomp source that would apply lzma to the binary blob. There's no evidence it has ever been used, for the reasons I've given.
Index Array. The array added to the httpd is done by modifying the source code prior to build. It adds an object named websRomPageIndex that contains this array of information about the files in the concatenated binary blob.
The array consists of file names and sizes. An older version of webcomp included offsets as well, but these are redundant since the binary blob's files are concatenated in the same order as they appear in the index array. Therefore, as one walks the array, the size of each file gives a relative offset to the next file in the binary blob.
Craig Heffner's webdecomp, which can decompile and recompile webcomp'd web packages, is much better formatted code than webcomp itself, so I'll use its data structures, amended with my own comments.
Each member of the array is defined as:
ORIGINAL Format
/* OLD webcomp file_entry data type */
struct file_entry
{
uint32_t name; /* virtual offset of asciiz file name in httpd */
uint32_t offset; /* offset to file in blob (unnecessary) */
uint32_t size; /* size of file data in blob */
};
NEW Format
/* NEWER webcomp file_entry data type */
struct new_file_entry
{
uint32_t name; /* virtual offset of asciiz file name in httpd */
uint32_t size; /* size of file data in blob */
};
DD-WRT has hacked on the webcomp code a few times. Their modifications are not publicly available. In fact, only the old webcomp.c using the original file_entry data type is included in the DD-WRT source code. They may have been responsible for the switch to the more efficient file_entry data type that excludes the redundant offsets.
They are certainly responsible for a recent addition that obfuscates all file sizes in the index array. They do this by adding a constant to the sizes. In the most recent builds of DD-WRT, this appears to be 0x4BF8. This can vary between builds.
Therefore, the obfuscation is quite simple:
new_file_entry.size += key;
Dynamically Calculating the Key. Since we can assume the real size of files within the package (many haven't changed in years, if ever), we can calculate the key used with a firmware. This has the added benefit of working on older images that had no key (0). We do this by finding the file we know the size of (index 0 is fine), then:
key = new_file_entry.size - assumed_known_size;
For an additional sanity check, one could confirm that the sample file is indeed extracted to match the known sample of that file. A simple hash compare would do the trick.
Update: See devttys0's answer that utilizes the sum of all files vs the actual blob size, then divides by the number of files to get the average difference, which is the key.
Created by Craig Heffner, this tool can extract and rebuild webcomp'd packages. It's, ironically, far superior code to the webcomp tool. I recently modified webdecomp to handle these new DD-WRT keys, calculating the key at extraction and saving it for the Later rebuild. This is part of the Firmware Mod Kit.
Small update to 90h's answer: the key can be calculated dynamically without making assumptions about the size of any specific file.
The total of all the file sizes reported in httpd should match the size of the /etc/www file, which is the actual size of all the files. Since the same fixed value is added to all the file sizes reported in httpd, you can simply take the difference in reported size vs actual size and divide it by the number of files to find what fixed value was used:
key = (total_reported_size - actual_etc_www_size) / number_of_files
The webdecomp tool didn't work for me on the latest builds of DD-WRT, managed to extract the firmware though.
root@ubuntu:~/Desktop/firmware-mod-kit/src/webcomp-tools$ ./webdecomp --httpd="/home/root/Desktop/firmware-mod-kit/fmk/rootfs/usr/sbin/httpd" -www="/home/root/Desktop/firmware-mod-kit/fmk/rootfs/etc/www" --dir="/home/root/Desktop/www" --extract
Failed to locate websRomPageIndex!
Failed to detect httpd settings!
Failed to process Web files!
root@ubuntu:~/Desktop/firmware-mod-kit/fmk/rootfs/usr/sbin$ file httpd
httpd: ELF 32-bit LSB executable, MIPS, MIPS32 version 1, dynamically linked, interpreter /lib/ld-uClibc.so.0, corrupted section header size
Looks like they have obfuscated the section headers.
root@ubuntu:~/Desktop/firmware-mod-kit/src/webcomp-tools$ readelf -a /home/root/Desktop/firmware-mod-kit/fmk/rootfs/usr/sbin/httpd
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 01 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 1
Type: EXEC (Executable file)
Machine: MIPS R3000
Version: 0x1
Entry point address: 0x401a30
Start of program headers: 52 (bytes into file)
Start of section headers: 0 (bytes into file)
Flags: 0x54001005, noreorder, cpic, o32, mips16, mips32
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 7
Size of section headers: 0 (bytes)
Number of section headers: 0
Section header string table index: 0
There are no sections in this file.
There are no sections to group in this file.
