How to break on instruction that hasn't been traced yet?

Question

I have some questions regarding IDA and Ollydbg.

First, is there a way to create a record with traced instructions and let the debugger automatically break, when it hits an instruction that hasn't been traced in the record ?

My second question is refering to IDA tracing options. I wrote a simple Win32 GUI application in C++ which mainly consists of a message pump (GetMessage, TranslateMessage and DispatchMessage) and a WindowProcedure. I tried the following:

1. I set a breakpoint in the message pump;
2. Started the tracing from there;
3. Set a second breakpoint in the WindowProcedure that triggers a break when I click on a button.

What IDA traced are only the instructions from the message pump not the ones from the WindowProcedure, though it should at least have traced the DefaultWindowProc instruction.

Can anyone tell me whats the reason for that?

Answer 1

1. Intel's Pin software can do this. It may take some effort, however.
2. Is WindowProcedure a system call? I don't think IDA can debug kernel code - you may have to use windbg if that is the case.

Answer 2

STEGANOGRAPHED IMAGE :) COPY HEXEDIT AND RIP

This answer is related to hittrace feature of ollydbg
Attached is a odbg110 plugin hitaid.dll
the dll has been 7zipped and hexedited to be uploaded here as jpg
download hexedit and delete first 10 bytes and save as hitaid.7z
use 7zip to extract hitaid.dll from the archieve

ollydbg has a feature called Hittrace this plugin leverages that feature to achieve what is queried

put the dll in odbg110 plugin path
open calc.exe in odbg
analyse the main module viz calc.exe
right click in disasm window select hittrace in context menu
select add all recognized procedures submenu
all recognized procedures will be marked with grey marker
run the exe with f9
all instruction that were hit would be colored red
hitting ctrl+L from start of disasm window will cycle through all hits
use plugin to remove all the hits
if the required feature say button 7 is clicked now
the new code path that pertains to button 7 would be colored red
new hits can be reached by selecting disasm window start and pressing ctrl+L
plugin can be used repeatedly to remove hits the app need not be paused for the interaction you can set f2 breakpoints as required in the new code path and reenable hittrace via disasm context menu

src below

#include <windows.h>
#include "plugin.h"
#pragma warning(disable:4100)
BOOL WINAPI DllEntryPoint(HINSTANCE hi,DWORD reason,LPVOID reserved) 
{
    return 1;
};
extc int _export cdecl ODBG_Plugindata(char shortname[32]) 
{
    strcpy(shortname,"HitAid");
    return PLUGIN_VERSION;
};
extc int _export cdecl ODBG_Plugininit( int ollydbgversion,HWND hw,ulong *features) 
{
    Addtolist(0,0,"Hit Trace Aid Plugin by blabb"); 
    return 0;
};
extc int _export cdecl ODBG_Pluginmenu(int origin,char data[4096],void *item) 
{
    switch (origin)    
    {
    case PM_MAIN:
        strcpy(data,"0 &HitAid");
        return 1;
    default: break;
    };
    return 0;
};
extc void _export cdecl ODBG_Pluginaction(int origin,int action,void *item) 
{
    uchar *buff , *retu;
    unsigned long psize, i , addr;
    t_module * mod = Findmodule( Plugingetvalue(VAL_MAINBASE) );    
    addr = mod->codebase;
    if (origin==PM_MAIN)    
    {
        switch (action) 
        {
        case 0:
            retu = Findhittrace ( addr, &buff,&psize);
            for (i=0;i < psize;i++ )    {
                if ( retu[i] != ( TR_TRACE | TR_HIT )  ) 
                {
                    continue;
                }
                Modifyhittrace((addr+i) , (addr+i+1),ATR_REMOVE);
            }
            break;
        default:
            break;
        };
    };
};

compiled with vs 2010 express and modified pdk by alex downloadable from www.ollydbg.de

cl /nologo /W4 /O1 /J /analyze /LD /WX /D"_CRT_SECURE_NO_WARNINGS" hitaid.cpp    
/link /subsystem:windows /dll /ENTRY:DllEntryPoint/MERGE:.rdata=.text .\ollydbgvc7.lib