The dialog box is a password challenge, and I'd like to catch/trace/observe the code (hopefully the password-checking code) that gets executed right after clicking the OK button of the dialog box.

I can't seem to find a way to do this in IDA. What's the correct sequence of actions in IDA to accomplish this?

5 Answers 5


This will probably be much easier to do if you use a debugger to dynamically analyze the process than with static analysis.

If the application uses the plain windows API, you could try a software like spy++ to get the address of the dialog procedure, which is called whenever something interesting happens within the dialog Or, you could place a breakpoint on then DialogBox API functions (there are several of them, DialogBoxA for ascii strings, DialogBoxW for wide char strings, and some DialogBoxEx variants). Once you know the address, you can look up that in IDA.

If the application uses some window toolkit (like gtk, qt, swing if it's a java application, whatever), this might be a bit harder, because these toolkits will probably use only a small subset of the Windows API and do the event handling themselves. It might, however, still be successful if you place a breakpoint on GetWindowText and/or GetDlgItemText (again, in several A/W/Ex variants) to check when and where the application actually reads what you've entered.

If, for some reason, you absolutely can't use a debugger and have to use IDA only, you can still try to cross-reference the above functions and work your way from there. But, since you don't know which of the cross-references is used in the password checking code, this will probably be much harder/more time consuming.

Apart from this, some of the ideas in my answer to a similar question might help you.


Initially, you could simply search for any strings which are printed in the dialog box, and the code which is executed after user input should be easy to locate thereafter, providing no obfuscation has been employed. For example, in IDA Pro:

enter image description here

If it's a simple password crackme, you should be looking for code which iterates through each character of a string, i.e. the user input, and compares it to some hardcoded value. Alternatively, as others have elaborated, you could look for specific uses of the Win32 API and find cross-references.

  • 1
    The problem with this approach is, if it's a windows dialog, it'll be defined in some resource in the exe file, not in code/data, and you won't find the strings anywhere in IDA. Commented Jul 31, 2014 at 17:43
  • Although you can load rsrc segment in IDA if I remember right? Commented Aug 7, 2014 at 15:14
  • 1
    @DominikAntal: Yes, before the auto-analysis, IDA provides a 'load resource segment' checkbox option.
    – JamalS
    Commented Aug 7, 2014 at 15:14

I'd run the target and let it to show the dialog box, then I'd attach it to debugger. I'd break into the debugger, switching between the threads till I see which one called dialog box. If found I'd print the call stack.

Static analysis may be difficult if the dialog box is called via an address calculated runtime (virtual function) but you can always try the method above.


Not sure which of the above answers I should accept. I'll just report here what worked for me, using just IDA:

  1. Since the main window and main menu already exist upon firing up the program, I looked for a breakpoint that has the password challenge dialog as its downstream. In this case, it was a FileOpen subroutine.
  2. Clicking the File -> Open menu item and choosing the password protected file causes the breakpoint to be hit, pausing the execution of the program.
  3. Keep on stepping over until the password challenge dialog box pops up.
  4. Backtrack by setting last executed instruction as breakpoint.
  5. Repeat steps 2 to 4 until the code firing up the dialog box is identified.

Checking for strings such as "check", "password" and other prospective keywords didn't work as they're nowhere to be found.

Luckily, in the course of the process, a subroutine labeled as "chk_pass" was encountered, and I was able to extract the password from the EDX register at which the password was momentarily stored.


Sometimes you'll see a RegisterClass(Ex) call with a pointer to a structure which includes a function pointer to a function which processes events. That callback is where you usually want to look, specifically for a Msg of WM_COMMAND and a wParam of 0xC8.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.