How did this 80286 detection code work?

Question

I'm poking through a disassembled 16-bit DOS game circa 1992. The original system requirements state that the game needs an IBM AT-compatible machine or later, with the 286 processor, to run. And there's a stub around main() that checks for the processor and displays an error message if one is not found.

I was intrigued as to what was actually being tested for, and I tracked down what appears to be the test procedure. It's comprised of five sub-tests, which are run conditionally, and it returns an integer in the range 0..7 depending on the results of the tests. I figured out, broadly, what the code does (although there may be errors; I'm still rather inexperienced and sometimes misread/misinterpret the meanings of instruction sequences).

; ... stack setup omitted ...
pushfw

; ==========================================
; === CHECK #1 =============================
; ==========================================
; Sets FLAGS to 0x0 and then immediately reads it back. On an 8086/80186, bits
; 12-15 always come back set. On a 80286+ this is not the case.
; 8086/80186 behavior: jump to check 3.
; 80286+ behavior: fall through to check 2.
xor ax,ax      ; AX=0x0
push ax
popfw          ; pop 0x0 into FLAGS
pushfw
pop ax         ; pop FLAGS into AX

and ax,0xf000  ; bits 12-13: IOPL, always 1 on 86/186
cmp ax,0xf000  ; bit 14: NT, always 1 on 86/186
               ; bit 15: Reserved, always 1 on 86/186, always 0 on 286+
jz check3

; ==========================================
; === CHECK #2 =============================
; ==========================================
; Only runs if CPU is plausibly an 80286. Last check before returning.
; Sets DL=0x6 if IOPL and NT flag bits are all clear.
; Sets DL=0x7 if any bits in IOPL/NT flags are set.
mov dl,0x6     ; DL is the proc's return val
mov ax,0x7000
push ax
popfw          ; pop 0x7000 into FLAGS
pushfw
pop ax         ; pop FLAGS into AX

and ax,0x7000  ; bits 12-13: IOPL
               ; bit 14: NT
jz done
inc dl         ; DL=0x7 if any bit was set
jmp done
nop

; ==========================================
; === CHECK #3 =============================
; ==========================================
; Only runs if CPU seems to be an 8086/80186.
; Sets DL=0x4 and moves on to...
;   check 4 if 0xff >> 21 == 0
;   check 5 otherwise (how can this happen?)
check3:
mov dl,0x4     ; DL is the proc's return val
mov al,0xff
mov cl,0x21
shr al,cl      ; AL = 0xff >> 0x21
jnz check5     ; when does this happen?

; ==========================================
; === CHECK #4 =============================
; ==========================================
; At this point, DF is still 0. ES doesn't
; point to anything sensible.
; Sets DL=0x2 if the loop completes.
; Sets DL=0x0 if the loop does not complete.
; Moves onto check 5 unconditionally.
mov dl,0x2     ; DL is the proc's return val
sti            ; are interrupts important?
push si
mov si,0x0
mov cx,0xffff
rep lods [BYTE PTR es:si] ; read 64K, ES[SI]->AL, all junk?
pop si
or cx,cx       ; test if loop reached 0
jz check5
mov dl,0x0     ; didn't hit 0. interrupted?

; ==========================================
; === CHECK #5 =============================
; ==========================================
; Leaving memory addresses here because they seem important.
; Here, DL is either 0x0 or 0x2 from check 4, or 0x4 from check 3. Looks like,
; contingent on the INC instruction getting overwritten, DL either stays at
; 0x0/0x2/0x4, or becomes 0x1/0x3/0x5.
check5:
00000B74  push cs
00000B75  pop es        ; Set ES to CS. (why not mov es,cs? illegal?)
00000B76  std           ; DF=1, rep decrements CX
00000B77  mov di,0xb88
00000B7A  mov al,0xfb   ; is this just an STI opcode?
00000B7C  mov cx,0x3
00000B7F  cli           ; are interrupts undesired?
00000B80  rep stosb     ; write 3 bytes, AL->ES[DI]
00000B82  cld           ; DF=0, why does it matter now?
00000B83  nop
00000B84  nop
00000B85  nop
00000B86  inc dx        ; destination when CX=1. overwritten?
00000B87  nop           ; destination when CX=2
00000B88  sti           ; destination when CX=3

done:
popfw
xor dh,dh      ; only keep low bits
mov ax,dx      ; return through AX
; ... stack teardown omitted ...
retf

; Return values:
; AX == 0x0: 8086, normal right-shift, loop aborted, overwrites
; AX == 0x1: 8086, normal right-shift, loop aborted, did not overwrite
; AX == 0x2: 8086, normal right-shift, loop finished, overwrites
; AX == 0x3: 8086, normal right-shift, loop finished, did not overwrite
; AX == 0x4: 8086, weird right-shift, overwrites
; AX == 0x5: 8086, weird right-shift, did not overwrite
; AX == 0x6: 286, with clear IOPL/NT flags
; AX == 0x7: 286, with set IOPL/NT flags

Here's what I can figure so far:

Check 1: Seems straightforward. Explicitly set FLAGS to 0x0 and then read it back. The 8086 will force all bits 12..15 to 1, and the 286 won't. Source.

Check 2: Only for the 286, seems to be similar to check 1 but with special focus on the protected mode flags. Not sure what significance this is to the caller.

(An aside: If we're assuming the CPU is a 286, couldn't it have been push 0x7000 instead of mov ax,0x7000; push ax?)

Check 3: Computes 0xff >> 0x21 and looks for a result other than 0. How does this ever happen? Is there a reason the nonzero result obviates the need for check 4?

Check 4: Reads 64K from ES into AL. Seems like busywork; ES has not been set to anything useful, and AL is not read from. Core of the test seems to be built around the idea of CX never reaching zero, possibly because of an interrupt somewhere during the loop? Shouldn't the interrupt procedure iret and return here to finish?

Check 5: Self-modifying code? Looks like it replaces the last few instructions of the test with STI, thus removing the INC that would otherwise affect the return value? What is the circumstance under which it would fail to overwrite, and thus execute the INC?

(An aside: Could push cs; pop es be rewritten as mov es,cs or is that not a legal form?)

I feel like I'm pretty far along in understanding it, but there are clearly a few holes remaining. I'm also not anywhere near fluent in x86, so there may be misinterpretations in my translated comments as well. I get the sense that there's some real cleverness here, written by somebody who knew the intricacies of these machines at a very detailed level. I'd like to understand their magic on some level, if I can.

Answer 1

I'll dive into the past and try to give an explanation of the different checks you are observing in your software. I found three references explaining the behaviour (as I hope), which I will reference to in this answer as /1/, /2/, /3/.

/1/ http://www.drdobbs.com/embedded-systems/processor-detection-schemes/184409011

It is an archive article from the DrDobbs Journal (very regrettably not existing any more since many years, but their archive still being a valuable resource), by Richard Leinecker, June 01, 1993, called "Processor Detection Schemes".

/2/ https://github.com/lkundrak/dev86/blob/master/libc/misc/cputype.c

It is a program written by Robert de Bath, published Oct 23, 2013, also covering problems like the one here in considerable detail, unfortunately not with too much of code comments.

/3/ iAPX 86/88, 186/188 User's Manual, Programmer's Reference, intel, May 1983

It is the Intel Programmers Reference for the processors listed in the title, still valid in many aspects (a nice example of the extremely fast pace of technology change in some fields...).

Your CHECKs:

CHECK1: You gave the explanation yourself. It may be verified in /1/, LISTING ONE (contained in the article as well). I will not reproduce the code here and not comment further, as there is nothing to add to your explanation.

CHECK2: Checks whether the processor is a 286 or higher (like 386 or 486). I'll quote the description of /1/, together with their code. Quote:

; Is It an 80286?
; Determines whether processor is a 286 or higher. Going into subroutine ax = 2
; If the processor is a 386 or higher, ax will be 3 before returning. The
; method is to set ax to 7000h which represent the 386/486 NT and IOPL bits
; This value is pushed onto the stack and popped into the flags (with popf).
; The flags are then pushed back onto the stack (with pushf). Only a 386 or 486
; will keep the 7000h bits set. If it's a 286, those bits aren't defined and
; when the flags are pushed onto stack these bits will be 0. Now, when ax is
; popped these bits can be checked. If they're set, we have a 386 or 486.
IsItA286    proc
        pushf               ; Preserve the flags
        mov ax,7000h        ; Set the NT and IOPL flag
                            ; bits only available for
                            ; 386 processors and above
        push    ax          ; push ax so we can pop 7000h
                            ; into the flag register
        popf                ; pop 7000h off of the stack
        pushf               ; push the flags back on
        pop ax              ; get the pushed flags
                            ; into ax
        and ah,70h          ; see if the NT and IOPL
                            ; flags are still set
        mov ax,2            ; set ax to the 286 value
        jz  YesItIsA286     ; If NT and IOPL not set
                            ; it's a 286
        inc ax              ; ax now is 4 to indicate
                            ; 386 or higher
YesItIsA286:
        popf                ; Restore the flags

        ret                 ; Return to caller
IsItA286    endp

I hope you'll see the similarity to your code immediately.

CHECK3: Determines whether you have a 80186/80188 or earlier. Quote from /3/, p.3-26, chapter "SHIFTS":

"On the 8086,88 up to 255 shifts may be performed. ...

... Before the 80186, 188 perform a shift (or rotate) they AND the value to be shifted with 1FH, thus limiting the number of shifts occurring to 32 bits."

Your code, commented:

mov dl, 0x4     ; DL is the proc's return val
mov al, 0xff    ; al contains 0xff
mov cl, 0x21    ; According to the above explanation from Intel,
                ; this value in cl is in an 80186/188 converted to 1, by ANDing with 0x1F.
shr al, cl      ; 80186/188 => al = 0x7F
                ; other: al = 0
jnz check5      ; goto check5 if you have an 80186/188

CHECK4: This one is not quite clear. However, it seems that it is the test of some CPU error. It seems to test for the CMOS version of an 8086/88.

/2/ lists the following code with comment, starting from line 271ff:

; The CMOS 8088/6 had the bug with rep lods repaired.
cmos:   push si
    sti
    mov cx, #$FFFF
rep
    lodsb
    pop si
    or cx,cx
    jne test8
    mov bx,#2   ; Intel 80C88

It's not exactly your code, but very similar, thus I assume your code as well tests for the 80C88 processor. I had never heard about this bug, and found no further information of it in the web. Thus, kind of a guess.

CHECK5: This one tests whether we have an 8086/80186 or an 8088/80188, i.e. 16-bit or 8-bit machine. And your suspicion was right, it is self-modifying code. The idea is whether the self-modified instruction has already been in the prefetch queue or not. This check is covered in /1/ and /2/ as well. I reproduce the comment from /1/.

The author in /1/ describes it like this:

"Differentiating between 8088s and 8086s is trickier. The easiest way I've found to do it is to modify code that's five bytes ahead of IP. Since the prefetch queue of an 8088 is four bytes and the prefetch queue of an 8086 is six bytes, an instruction five bytes ahead of IP won't have any effect on an 8086 the first time around."

As a reference, Intel writes in its manual /3/, p.3-2 "Bus Interface Unit":

"The 8088/188 istruction queue holds up to four bytes of instruction stream, while the 8086/80186 queue can store up to six instruction bytes."

I won't reproduce the code from /1/ here (being very similar), but rather re-comment your code with some remarks which I hope explain the case.

check5:
00000B74  push cs
00000B75  pop es        ; Set ES to CS. (why not mov es,cs? /1/ uses mov ax, cs, mov es, ax)
00000B76  std           ; Cause stosb to count backwards (di is decremented)
00000B77  mov di,0xb88  ; di==offset of code tail to modify
00000B7A  mov al,0xfb   ; is this just an STI opcode? ;IMO yes, /1/ uses 0x90 al==nop instruction opcode
00000B7C  mov cx,0x3    ; Set for 3 repetitions
00000B7F  cli           ; are interrupts undesired? Yes, I remember having read somewhere (no quote though) 
                        ; that the next instruction can be interrupted, without the cli. 
                        ; This of course would spoil the trick.
00000B80  rep stosb     ; write 3 bytes, backwards from Addr 0xb88
                        ; !!! 5 bytes down is the critical instruction 
                        ; which will be either already in the queue (8086/186) or not (8088/188)
00000B82  cld           ; Clear the direction flag
00000B83  nop          ; Three nops in a row
00000B84  nop          ; provide dummy instructions
00000B85  nop
00000B86  inc dx        ; <<<=== This instruction is executed ONLY in the 8086/186 case. 
                        ; In the 80188/88 case, it is overwritten with STI
00000B87  nop           ; dummy instruction
00000B88  sti           

As your return register dx came with a value of 4 from CHECK3, it will after CHECK5 have a value of 5 in the 16-Bit case.

Answer 2

Check 2: While check 1 tests if the high-order bits of the flag word can be cleared, check 2 tests whether they can be set. On an 80286, these bits cannot be set in real mode, while on an 80386 they can.

Check 3: This is testing what kind of shifter the processor has. Some (the newer ones) have a barrel shifter that effectively masks the shift count to the word size (and the use of 0x21 as the shift count suggests to me that the difference appeared in the post-80286 era). So a shift by 0x21 (33) gives the same result as a shift by 33 - 32 = 1. I don't know at which generation the barrel shifter appeared.

Check 4: I can't remember the details, but parts of it seem familiar to me. It's either got something to do with the repeat count being wrong after a maximal-length loop, or something with having a double instruction prefix that triggers a CPU bug. I think it's the latter, and the order of prefixes matters. When an interrupt handler returns, the instruction pointer is set to the wrong address and one or more prefixes are forgotten. Illustration: https://www.youtube.com/watch?v=6FC-tcwMBnU Note that the code you have here actually has the es: override prefix first, so the loop should always complete! Could this be a CPU bug detection routine, that itself contains a bug?

Check 5: This is checking for an instruction cache that operates independently of any data cache. On an 80486 you can stomp all over the 16-byte window in which the processor is currently executing, and it will still execute the old contents that were loaded into the (separate) instruction cache. I think Pentium+ processors detect this overwriting and flush the instruction cache and prefetch queue. Even the earliest x86 processors have a prefetch queue long enough (except the 8088) to cover the instructions being overwritten. Conditions under which the new code gets executed: on a Pentium+ (IIRC), under a single-stepping debugger, in v86 mode where the CLI instruction doesn't really take effect, and an interrupt occurs.