Windows PE64 throw error while allocating heap for unknown reason?

Question

For some reason i'm not aware of ( some heap allocation limit ) while allocating memory for some classes, an application ( win64 ) raises this exception:

DumpedMini Result[0] Exception Code[EXCEPTION_ACCESS_VIOLATION]
Fatal: Man.exe caused an EXCEPTION_ACCESS_VIOLATION(ThdId:10172) in module WApi.dll WApi::COdbcCommand::COdbcCommand(), Source(Wapicodbc.inl:341)
7FF9ED373847: The instruction at 0x7FF9ED373847 referenced memory at 0xFFFFFFFFFFFFFFFF. The memory could not be read (exc.code c0000005, tid 12420)
    :00007FF9ED373847 movaps  xmmword ptr [rcx+200h], xmm6
    00007FF9ED373847    ntdll.dll   ntdll_RtlCaptureStackContext+1E457

I don't have any access to application source , i debug it in ida , also there is similar application that does has same logic (allocating exactly same classes, but it doesn't throw error's and allocating up to 1gb in memory, while this app is stuck on 109k)

What tools or techniques i can use , to analyze what is behind the memory limiting , or what problem it could be ?

Answer 1

It seems like you have an issue with dereferencing unallocated memory. I don't think that it's related to the heap in any way. It looks like the pointer address that is resolved by ptr [rcx+200h] is in fact points to 0xFFFFFFFFFFFFFFFF address, and when trying to access this memory, the application crashes, because this page is not allocated.

In order to investigate the issue, try to understand what is the origin of the bad ecx register value that is used at 00007FF9ED373847 - just follow in the disassembly where is it defined, and what goes wrong in your specific scenario - for example, maybe it's origin a function call, that returned an error, but the return value wasn't checked against error codes.