Decoding the wire protocol (over RS232) for custom hardware <-> windows application?

Question

Background

I have a hardware device with an RS232 port, and a Windows (XP) application that it talks to. No OS-level drivers beyond basic serial, so all the decoding is done in the app.

I have the ability to capture bidirectional wire traffic and generally snoop on the serial port (good ol' Sysinternals Portmon to logfile, with some perl post-processing (parsing out IRP_MJ_(READ|WRITE) events) to get raw byte dumps).

From this I've gotten the basic wire protocol details (baud rate, settings, etc)

Messages appear to be in the format { $body }\n, where sent commands are usually a single byte or 2 in the body, responses are much more complex.

They span several messages, and appear to have additional structure, something like:

• Header ({IDCOMPLIANCE ...})
• Metadata? ({SA...})
• Data[+] ({DL20...})
• End of Data Indicator ({TE5186})

Actual example follows:

> {C}\x0D
< {IDCOMPLIANCE\x20\x20\x20\x20\x20\x20D245}
{SA001FD8BL0001C061BC}
{DL20I000000V80050300DE076A0012000400AF002400FF030300DE0700000000010003000000B883}
{DL20I000020VFF140200DE070900010001000D000100FF0C0200DE076E006E000300A5075904C953}
{DL20I000040VFF0B0200DE07120111010700DE11E406FF0A0200DE070401030105008D12640B3B0C}
{DL20I000060VFF090200DE07F400F400050046112F0AFF080200DE076C016C0107003419950D7181}
{DL20I000080VFF070200DE0753015101060063172A0CFF060200DE07D400D3000A00BE0DF504877D}
{DL20I0000A0VFF050200DE07F200F200040047117309FF040200DE071601150107000414F10C292B}
{DL20I0000C0VFF030200DE07D400D4000500260FC108FF020200DE07B900B90006004C0D26083825}
{DL20I0000E0VFF010200DE07B601B60107001E21F718FF1F0100DE0710010C010900B913150DE89C}
{DL20I000100VFF1E0100DE0785017F010900531A2F0EFF1D0100DE071B01180108001C146D0CE542}
{DL20I000120VFF1C0100DE077E007B0005009C092107FF090100DE0754000100020008000100A748}
{DL20I000140V0000000000000000000000000000000000000000000000000000000000000000D71C}
{DL20I000160V0000000000000000000000000000000000000000000000000000000000000000725F}
{DL20I000180V000000000000000000000000000000000000000000000000000000000000000010A5}
{DL20I0001A0V00000000000000000000000000000000000000000000000000000000000000004CC1}
{TE5186}

I also have access to the controlling application which I can get to log certain data (which may or may not be quite what it's talking over the wire) to friendly formats like CSV etc.

Question

Given my progress so far is mostly matching up CSV outputs with wire dumps, is there any value in trying to investigate the actual application more...invasively?

I have very little experience with asm, windows binaries, and windows debugging, but it seems like it might be possible to poke around in the binary either statically or while it's running, and look for where the decoding happens.

If I can find that, I can hopefully scrabble up enough asm to understand how it's being generated/parsed, and map that to what I see it actually sending.

The problem is actually finding those [de]coding routines with my limited knowledge.

Two approaches suggest themselves to me:

1. Run inside a debugger, figure out how to set a breakpoint on serial port read/write, and step through from there looking for decoding logic.

2. Load into a decompiler, and either (a) trace calls around serial read/writes, or (b) look for known strings that appear in the output CSVs, and work backwards to the code that's using them.

[Note that I'm doing this in a VMWare VM on a Mac, in case that matters for anything]

I've currently been playing around with OllyDbg for windows, and the demo version of Hopper Decompiler, which is affordable if it'll be of use.

IDA or Hex-Rays would be nice, but is a bit out of my budget unless there's something there (like, a 'Decode Mystery Protocol' button) that would justify it.

So,

• Does anyone recognise the format described above?

• Are there better tools for RS232 (actually via USB-serial adapter) sniffing? I found USBpcap (hxxp://desowin.org/usbpcap/) which can generate wireshark traces, but digging though it for the actual serial data was tedious, and capture isn't realtime.

• What would people suggest my next moves be (keep hacking at black-box csv/wire-data, decompile, or debug?)

• If debug, is ollydbg capable of breakpointing on serial port read (and where might I find n00b-level docs on how to do so?)

• If decompile, what approach should I take (forward from port read, or back from csv/strings? Again, pointers to how to do that would be great.)

• If black-box, any inspiration as to the structure of the {DL20...} messages in particular?

I suspect it breaks down something like:

• DL20 - not sure what the 20 indicates. Length of some kind?

• I000020 - clearly some sort of index/address

• V - Marker for 'vector'/array data to follow?

• FF140200DE070900010001000D000100 Entry #1

• FF0C0200DE076E006E000300A5075904 Entry #2

Each entry contains a date (dd/mm/yy, probably in a weird format like 'fractional number of fortnights since first summer solstice following the Fall of Rome'), and a bunch of integer and fractional (fixed point rather than float, probably)

Some fields are I think 16 bit, and little-endian (Entry#2 last 4 bytes is 0x0459 which matches observed data)

• C953 - Not sure, maybe checksum or CRC? Tried a few obvious standards taht don't match it though, and might be over the whole packet or just the data.

---

[Apologies for the broken links, need more rep to add >2 apparently]

Answer 1

My guess:

DL20I000000V80050300DE076A0012000400AF002400FF030300DE0700000000010003000000B883
^^  ^      ^                                                                ^^^^
||  Addr   Data                                                            CRC16
|` Length
` Data tag

• D for data
• L20 for 0x20 or 32 hex bytes, or 64 hex digits
• I000000 the address (note it increases by 0x20 per row, supporting both L20 as length and I as address)
• V the data, size matching L
• And a 16-bit CRC or checksum, supported by the final rows which are "all 0" except each with a unique checksum, suggesting the address is part of the formula.

It looks vaguely like a flash file format for programming PROMs or microcontrollers. IHEX doesn't have letters. SREC is similar to what you have.

Answer 2

Netzob is a tool expressly built for this purpose. I have nothing to do with the creation of the tool, and it does still have bugs in my experience, but it's very useful for this kind of reverse engineering of protocols.