The dialog box is a password challenge, and I'd like to catch/trace/observe the code (hopefully the password-checking code) that gets executed right after clicking the OK button of the dialog box.
I can't seem to find a way to do this in IDA. What's the correct sequence of actions in IDA to accomplish this?
This will probably be much easier to do if you use a debugger to dynamically analyze the process than with static analysis.
If the application uses the plain windows API, you could try a software like spy++ to get the address of the dialog procedure, which is called whenever something interesting happens within the dialog Or, you could place a breakpoint on then DialogBox API functions (there are several of them, DialogBoxA for ascii strings, DialogBoxW for wide char strings, and some DialogBoxEx variants). Once you know the address, you can look up that in IDA.
If the application uses some window toolkit (like gtk, qt, swing if it's a java application, whatever), this might be a bit harder, because these toolkits will probably use only a small subset of the Windows API and do the event handling themselves. It might, however, still be successful if you place a breakpoint on GetWindowText and/or GetDlgItemText (again, in several A/W/Ex variants) to check when and where the application actually reads what you've entered.
If, for some reason, you absolutely can't use a debugger and have to use IDA only, you can still try to cross-reference the above functions and work your way from there. But, since you don't know which of the cross-references is used in the password checking code, this will probably be much harder/more time consuming.
Apart from this, some of the ideas in my answer to a similar question might help you.
Initially, you could simply search for any strings which are printed in the dialog box, and the code which is executed after user input should be easy to locate thereafter, providing no obfuscation has been employed. For example, in IDA Pro:
If it's a simple password crackme, you should be looking for code which iterates through each character of a string, i.e. the user input, and compares it to some hardcoded value. Alternatively, as others have elaborated, you could look for specific uses of the Win32 API and find cross-references.
I'd run the target and let it to show the dialog box, then I'd attach it to debugger. I'd break into the debugger, switching between the threads till I see which one called dialog box. If found I'd print the call stack.
Static analysis may be difficult if the dialog box is called via an address calculated runtime (virtual function) but you can always try the method above.
Not sure which of the above answers I should accept. I'll just report here what worked for me, using just IDA:
Checking for strings such as "check", "password" and other prospective keywords didn't work as they're nowhere to be found.
Luckily, in the course of the process, a subroutine labeled as "chk_pass" was encountered, and I was able to extract the password from the EDX register at which the password was momentarily stored.
Sometimes you'll see a RegisterClass(Ex) call with a pointer to a structure which includes a function pointer to a function which processes events. That callback is where you usually want to look, specifically for a Msg of WM_COMMAND and a wParam of 0xC8.
