Reverse engineering the IBM BSC (Bisync) protocol

Question

I am trying to connect to an old piece of communication controller which use the IBM BSC synchronous protocol but I have problems to get the CRC right.

Background

The protocol itself is described quite well in this document:

http://bitsavers.trailing-edge.com/pdf/ibm/datacomm/GA27-3004-2_General_Information_Binary_Synchronous_Communications_Oct70.pdf

The sending communication processor is using the Motorola MC6852 chip. But the chip doesn't have hardware crc circuitry so there are software inside the comm. processor that does crc.

The actual communication controller is described here: http://storage.datormuseum.se/u/96935524/Datormusuem/Alfaskop/Alfaskop_System_41_Reference_Manual_IBM3270_Emulation.pdf

Page 89 and onwards describes its use of BSC.

Then since this piece of equipment is IBM 3274 model C compatible this manual is applicable: http://bitsavers.informatik.uni-stuttgart.de/pdf/ibm/3274/GA23-0061-1_3274_Control_Unit_Description_and_Programmers_Guide_Jan84.pdf

Page 159 and onwards has essentially the same information as the other document.

Actual Messages

I have captured two messages sent by actual communication processor. Please note that the messages are coded in EBCDIC and not ASCII. These are responses to a POLL message. Page 172 and 175 in the IBM document above:

32 01 6C D9 02 40 40 40 70 03 26 88 and 32 01 6C D9 02 40 C8 40 50 03 0D 28

From my reading of the IBM document the CRC algorithm will reset when it sees a 01 (SOH) or 02 (STX) and then accumulate until it sees a 03 (ETX). A SOH followed by a STX will not reset CRC again at the STX, though. Essentially this means that the CRC bytes in the messages above are the 2688 and 0D28. There is a leading SYN (32) but that is not included in the CRC calculation since it is before the SOH character. The message used for CRC calculation is then: 6C D9 02 40 40 40 70 03 and 6C D9 02 40 C8 40 50 03 respectively. The SOH is not part of the calculation, however the trailing ETX just before the CRC is part of the CRC calculation.

The messages above are received by an Intel 8274 chip. No CRC checking was done in the Intel chip though.

Since the Intel 8274 chip do include a CRC checker/generator I should be able to both generate correctly formatted messages and to receive messages for further checking and investigation. I will pursue this path a bit to see if I could both receive and send using the 8274 chip and what CRC values that are generated.

The 8274 chip itself supports two CRC algorithms. CRC-16 and CCITT CRC-16. My understanding is that IBM used CRC-16 with the polynomial X^16+X^15+X^2+1. I.e 8005. What initial value for crc was used is not described anywhere what I can see. A good guess though would be 0000h or possibly ffffh.

The situation is that I would like to use a small STM32 micro controller to handle the BSC communications. I successfully had the program to achieve sync with the incoming data and extracted correctly formatted messages. But obviously communication would not work unless I can get the CRC calculation correct. WIP: https://github.com/MattisLind/alfaskop_emu/tree/master/Utils/BSCGateway

CRC reveng troubles

I have tried crc reveng tool to calculate CRC digits but I cannot get a match with the data above. Neither can I use crc reveng to search for an algorithm. It constantly reports that no models where found.

Generating CRC using crc reveng with above messages as data in and polynomial 8005 doesn't give the corresponding output CRC data. I tried several initial values for CRC and also tested various variants of options for bit order but no match.

Then I tried a few CRC algorithms in C found on internet. They all gave the same CRC value on the above messages but non of them matched the values in the messages neither the output from crc reveng.

It is very likely that I have done something wrong when applying crc reveng, but I cannot figure out what.

Another test of crc reveng

Just to get a better understanding of crc reveng I tried a sample message where the input data buffer, initial CRC value and output CRC value was well known: https://stackoverflow.com/questions/23638939/crc-16-ibm-reverse-lookup-in-c

The code in the first answer by Mark Adler give a result which is matching what is mentioned in the Maxim article. But I cannot recreate it in crc reveng, unfortunately.

Hints on what I am doing wrong, please!

Answer 1

If you have known data and a known CRC, and an unknown value (like the initial CRC value or the final XOR value), you can always loop through all possible values and find those that give you the answer you want. It might not be practical for 32-bit CRCs, but for 8- and 16-bit CRCs it shouldn't take too long to run.

You'll need at least two sets of known data to get more accurate values though, as many of the possible 65,536 values will give the right answer for a single message. But with two, very few values will produce the right CRC for both messages. Just generate both CRCs in a loop, and if both produce the right result then print the values used and continue with the next ones. With luck, at the end you'll have the correct constants used by that implementation - assuming the rest of the algorithm is correct of course!

Also, make sure your CRC implementation is handling word sizes correctly, e.g. a 16-bit implementation on a 32-bit architecture could accidentally be using bits above 15 that were just lost on the original 16-bit architecture.

I haven't used CRC RevEng before, but it sounds like this is what it does, so I'm not sure why you are getting no results. In your case the algorithm seems to be pretty well defined, but I just reverse engineered an algorithm that was documented as being CRC-8, but actually it seemed more like a checksum rather than a full on CRC algorithm, and was 16-bit not 8-bit, so you can't always trust the documentation! Possibly if the algorithm is nonstandard in some way this is why CRC RevEng can't pick it up.

Answer 2

SOLUTION

I got help from a guy named Peter. He gave me a piece of test code in C. The CRC algorithm looked pretty similar to the ones I already tried. But what was important was that he pointed out that the first sample message most likely had a bit error.

#include <stdio.h>

int crc16(unsigned char *ptr, int count)
{
    unsigned int crc;
    char i;

    crc = 0x0000;
    while (--count >= 0)
    {
        crc = crc ^ (unsigned int) *ptr++;
        i = 8;
        do
        {
            if (crc & 0x0001)
                crc = (crc >> 1) ^ 0xA001;  /* 0x8005 bit reversed */
            else
                crc = (crc >> 1);
        } while(--i);
    }
    return (crc);
}

void main()
{                   
                     /* 32  01  6C  D9  02  40  40  40  70  03  26  88 */

   unsigned char data1[] = {0x6c, 0xd9, 0x02, 0x40, 0x40, 0x40, 0x50, 0x03};

                     /* 32  01  6C  D9  02  40  C8  40  50  03  0D  28 */

   unsigned char data2[] = {0x6c, 0xd9, 0x02, 0x40, 0xc8, 0x40, 0x50, 0x03};

   printf("crc sent: 8826 computed: %4.4x\n", crc16(data1, sizeof(data1)));

   printf("crc sent: 280d computed: %4.4x\n", crc16(data2, sizeof(data2)));

   return;
}

And when testing the first corrected message and the second message using crc reveng it finds the algorithm

$ ./reveng -w 16  -s 6cd90240c84050030d28
./reveng: warning: you have only given 1 sample
./reveng: warning: to reduce false positives, give 4 or more samples
width=16  poly=0x8005  init=0x0000  refin=true  refout=true  xorout=0x0000  check=0xbb3d  residue=0x0000  name="CRC-16/ARC"
MattisMacBook:reveng-2.1.0 mattis$ ./reveng -w 16  -s 6CD90240404070032688
./reveng: warning: you have only given 1 sample
./reveng: warning: to reduce false positives, give 4 or more samples
./reveng: no models found
$ ./reveng -w 16  -s 6CD90240404050032688
./reveng: warning: you have only given 1 sample
./reveng: warning: to reduce false positives, give 4 or more samples
width=16  poly=0x8005  init=0x0000  refin=true  refout=true  xorout=0x0000  check=0xbb3d  residue=0x0000  name="CRC-16/ARC"

Then I tried getting crc reveng to do the same calculation as the code Peter provided above. It took a while to get the options right. For some reason I had to specify the polynom in the reverse bit order to make it work.

$ ./reveng -w 16 -P a001 -i 0000  -x 0000  -l  -d 
width=16  poly=0x8005  init=0x0000  refin=true  refout=true  xorout=0x0000  check=0xbb3d  residue=0x0000  name=(none)
$ ./reveng -w 16 -P a001 -i 0000  -x 0000  -l  -c 6CD9024040405003
2688

Conclusion and lessons learnt

I never thought there could be errors in received data. That was stupid. It is not maybe likely on 9600 bps connection but can happen. Then I was too focused on testing both messages at the same time and never saw that the second message actually returned OK CRC, although it had the bytes swapped.

So lessons for myself is:

1. Have more samples so that transmission errors can be spotted more easily
2. Don't assume that all messages are correct.
3. crc reveng has many options that at first sight can be hard to get right - keep trying!

Thanks to everyone that has helped!