5

I'm decompiling some Direct3D code that makes a lot of indirect calls to __stdcall functions.

For example:

call dword ptr [edx+0xC8h]

which is really:

pD3DDevice->SetRenderState();

IDA doesn't correctly guess the stack pointer change of these calls in every case, so I have to go through and Alt+K the correct SP value manually.

But after doing this I start running into a problem where one side of a branch will have the wrong SP value

enter image description here

I can Alt-K the first instruction with the erroneous SP value but this only takes effect on the next instruction.

Edit:

enter image description here

4
  • Can you show how it looks like in text mode (instead of graph)?
    – Igor Skochinsky
    Nov 16, 2020 at 7:27
  • 1
    @IgorSkochinsky Apologies for the extreme lateness of this reply (long story), but on the off-chance you're still willing to help, I've updated the OP with a text-mode screenshot which hopefully illustrates the problem.
    – quitegiddy
    Apr 17, 2023 at 11:47
  • the new screenshot doesn't include the code from the first one but some unrelated(?) function.
    – Igor Skochinsky
    Apr 24, 2023 at 15:18
  • 1
    @IgorSkochinsky Thanks for the response. I must have uploaded the wrong screenshot sorry about that. It's fixed it in the OP now. The right branch isn't out by quite as much as in the original screenshot for some reason. Taken from a fresh db with just the stack pointer corrected on the indirect calls and pD3DDevice named.
    – quitegiddy
    Apr 25, 2023 at 12:52

1 Answer 1

1

So, the problem is that you have two branches with different stack delta: E8 at 5018E4 (fall through from jnz) and F0 at 501c49 (destination of jnz). Normally they should be the same, however IDA failed to reconcile them, probably due to too many indirect calls in the function.

One peculiarity of manual stack delta adjustments is that they apply after the instruction at which you're setting them. In our case, we need to make 501C49 have the delta of E8, however we can't do it on the instruction itself but need to go to the previous one by address and not the control flow, i.e. the jmp at 501C44. Since that address has SPD=E0, specifying delta of -8 should work.

This is why such manipulations are best done in text view and not graph.

1
  • Thanks Igor, that's exactly what I was looking for. I must have been under the misconception that Alt-K would change the next instruction in the control flow rather than the next instruction in memory, which really wouldn't make any sense. Thanks again!
    – quitegiddy
    Jun 10, 2023 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.