I am analyzing a sequence of x86 instructions, and become confused with the following code:

135328495: sbb edx, edx
135328497: neg edx
135328499: test edx, edx
135328503: jz 0x810f31c

I understand that sbb equals to des = des - (src + CF), in other words, the first instruction somehow put -CF into edx. Then it negtive -CF into CF, and test whether CF equals to zero??

But note that jz checks flag ZF, not CF! So basically what is the above code sequence trying to do? This is a legal x86 instruction sequence, produced by g++ version 4.6.3.

  • 2
    post more instructions before the sbb
    – Igor Skochinsky
    Dec 8 '16 at 8:27
  • the test edx, edx will affect ZF. and about code it's hard to guess by a snipped assembly, but obviously the compiler is trying to save CF inside edx and then checking the edx (which is CF).
    – Arash
    Dec 8 '16 at 8:33
  • Cross-site duplicate: stackoverflow.com/questions/41031912/…
    – Cody Gray
    Dec 9 '16 at 17:08

The sbb edx, edx statement writes either 0 or -1 to edx, depending only on the value of the carry flag. The following neg edx simply reflects the value of the initial carry flag. Thus the jz in your sequence is nothing else than a jnc statement (jmp on non-carry). However, this sequence might be found with an additional, preceding neg eax. The neg statement clears the carry in the zero case, otherwise sets it. This sequence might be used as a test for true or false, depending whether edx has some arbitrary nonzero value (true) or zero value (false). The sequence with the additional neg would then look like this:

neg edx          ; clears the carry flag in the zero case, otherwise sets it
sbb edx, edx     ; if (cf == 0) then edx == 0, else edx == -1
neg edx          ; remains zero if initially edx has been zero, else 1
test edx, edx    ;
jz toSomewhere   ; jmp on edx having been zero initially

btw, this is one of the questions I am planning for a reversing quiz.

Have fun!

  • It is certainly a question that tests one's knowledge of the instruction set, but it's otherwise extremely questionable code. The entire thing could be replaced with a single JNC toSomewhere instruction... I cannot think of a good reason why a compiler would ever generate code like this.
    – Cody Gray
    Dec 9 '16 at 17:09
  • Of course you are right for the code without the neg in my first line. But with it, the compilers use such pattern frequently for the described true/false decisions. The MS compiler uses a slightly modified variant.
    – josh
    Dec 9 '16 at 17:56
  • Yes, MSVC uses the sbb+neg pattern (combined with a trailing or leading neg, inc, or dec as appropriate for the logic) extensively. But only when it is generating branchless code. If it's going to emit a conditional branch (in this case, jz) anyway, there's absolutely no point in paying the penalty for all of these additional instructions.
    – Cody Gray
    Dec 9 '16 at 17:58
  • Maybe the OP can unveil the secret by showing us some more of his code generated by g++...
    – josh
    Dec 9 '16 at 18:02
  • @CodyGray Hey, thank you so much for your very informative replies. I learned a lot for both here and the stackoverflow. The C++ code is actually from the botan project. I will try to find the corresponding source code for this instruction sequence and provide to you later. At the mean while, you can find the overall assembly code (the Botan RSA decryption example) at here. There are quite a lot of such instruction sequence in the disassembled code. Dec 9 '16 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.