ida_hexrays.lvar_t.stk.get_stkoff() incorrect ± 0x10

Question

When using the stack offset for a variable obtained via ida_hexrays.lvar_t.stk.get_stkoff() or ida_hexrays.vdloc_t.stkoff() the results vary between "correct", and 8 or 16 bytes out as measured either manually or as compared to the results obtained from:

id = idc.get_frame_id(here())
for member in idautils.StructMembers(id):
    offset, name, size = member

The incorrect offsets are sourced from:

func = idaapi.get_func(ea)
vu = idaapi.open_pseudocode(func.start_ea, 0)
[n.get_stkoff() for n in vu.cfunc.lvars if n.is_stk_var()]

[edit]: the difference appears to be caused by the pseudo-code vu reporting offsets relative to the minimal SPD, whilst everything else uses the post-prologue SPD (code beneath)

def GetPseudoStackOffsetCorrection(funcea):
    func = ida_funcs.get_func(funcea)
    return func.frsize + func.frregs + idc.get_spd(idc.get_min_spd_ea(func.start_ea))

[edit 2]: this alternate method works sometimes. it functions by looking at the insn located at lvar_t.defea (assumedly "defined at ea") and calculating from there. However 10% of the time lvar.defea points at a conditional jmp, so again -- very kludgy

def get_stkoff_from_lvar(lvar, debug=1):
    ea = idc.get_item_head(lvar.defea)
    func = ida_funcs.get_func(ea)
    if not func:
        return idc.BADADDR
    
    for n in range(2):
        if idc.get_operand_type(ea, n) == idc.o_displ:
            offset = idc.get_operand_value(ea, n) + func.frsize - func.fpd

            if debug:
                lvar_name = lvar.name
                sid = idc.get_frame_id(func.start_ea)
                frame_name = idc.get_member_name(sid, offset)
                print("[debug] offset:0x{:x}, lvar_name:{}, frame_name:{}"
                        .format(offset, lvar_name, frame_name))

            return offset 

[edit]: kludges aside (which I would prefer not to use) I have issues with my [...] "rename, retype and remap" (with regex) package, which I recently noted does not update the "stack" (visible in assembly) names when calling vu.rename_lvar as would normally happen when performing a rename via N

Without an accurate offset, there is no way for the code to rename the corresponding stack variable var_18 when renaming the pseudo-code variable v1 (for example).

Any solution to this general problem is welcome, though preferentially one that allows the correct stack location of pseudo-code variables would be best as I will shortly be attempting "watch-variables" via flare-emu.

These are some results from a test function written to compare the conflicting results I have been receiving, and to confirm to myself that there is no magic "just deduct func.frregs" type answer.

Function 1.

func.flags                  : FUNC_FRAME | FUNC_PURGED_OK | FUNC_SP_READY
func.frregs                 :   8
func.frsize                 :  c0
func.fpd                    :  a0
ida_frame.frame_off_retaddr :  c8
ida_frame.frame_off_lvars   :   0
ida_frame.frame_off_args    :  d0
ida_frame.frame_off_savregs :  c0

name           lvar_offset1 stk_offset 
-------------- ------------ ---------- 
range          0x30         0x28       
guide          0x38         0x30       
ImageBase      0x58         0x50       
_stack_padding 0xd8         0xd0    

Function 2

func.flags                  : FUNC_FRAME | FUNC_PURGED_OK | FUNC_SP_READY
func.frregs                 :   8
func.frsize                 : 1c0
func.fpd                    : 190
ida_frame.frame_off_retaddr : 1c8
ida_frame.frame_off_lvars   :   0
ida_frame.frame_off_args    : 1d0
ida_frame.frame_off_savregs : 1c0

name                      lvar_offset1 stk_offset 
------------------------- ------------ ---------- 
lpTopLevelExceptionFilter 0x50         0x40       
LibFileName               0x88         0x78       
Handle                    0xc8         0xb8       
lpProcName                0x110        0x100      
hObject                   0x198        0x188      
hModule                   0x1a0        0x190      

Function 3

func.flags                  : FUNC_FRAME | FUNC_PURGED_OK | FUNC_SP_READY
func.frregs                 :   8
func.frsize                 :  40
func.fpd                    :  20
ida_frame.frame_off_retaddr :  48
ida_frame.frame_off_lvars   :   0
ida_frame.frame_off_args    :  50
ida_frame.frame_off_savregs :  40

name   lvar_offset1 stk_offset 
------ ------------ ---------- 
accum1 0x20         0x20       
accum2 0x2c         0x2c 

I can provide the test code if required, though it's not small.

[edit: test code attached, warning: not pretty]

# test code: 
#     sync_lvars_to_stk(func_ea)

def get_func_flag_names(f):
    return [x for x in [k for k in dir(idc) 
            if k.startswith('FUNC_')] 
            if f.flags & getattr(idc, x)]

def _get_vu(ea, vu):
    if vu: return vu
    return idaapi.open_pseudocode(idaapi.get_func(ea).start_ea, 0)

def get_lvars(ea, vu=None):
    vu = _get_vu(ea, vu)
    return [n.get_stkoff() for n in vu.cfunc.lvars if n.is_stk_var()]

def dump_stkvars(ea=None, iteratee=None):
    def get_member_tinfo(sid, offset):
        s = ida_struct.get_struc(sid)
        m = ida_struct.get_member(s, offset)
        tif = ida_typeinf.tinfo_t()
        try:
            if ida_struct.get_member_tinfo(tif, m):
                return tif
        except TypeError:
            pass

    results = []
    sid = idc.get_frame_id(ea)
    for member in idautils.StructMembers(sid):
        o = AttrDict()
        o.offset, o.name, o.size = member
        o.mid     = idc.get_member_id(sid,    o.offset)
        o.name    = idc.get_member_name(sid,  o.offset)
        o.size    = idc.get_member_size(sid,  o.offset)
        o.flags   = idc.get_member_flag(sid,  o.offset)
        tif       = get_member_tinfo(sid,     o.offset)
        o.tifname = str(tif) if tif else ''
        o.sid     = sid
        if callable(iteratee): iteratee(o)
        results.append(o)
    return results

def indexBy(o, key):
    r = {}
    for x in o:
        r[x[key]] = x
    return r

def sync_lvars_to_stk(ea, vu=None):
    vu = _get_vu(ea, vu)
    func = idaapi.get_func(ea)
    print("\n{:28}: {}\n{:28}: {:3x}\n{:28}: {:3x}\n{:28}: {:3x}\n{:28}: {:3x}\n{:28}: {:3x}\n{:28}: {:3x}\n{:28}: {:3x}\n"
            .format(
                "func.flags",                  " | ".join(get_func_flag_names(func)),
                "func.frregs",                 func.frregs,
                "func.frsize",                 func.frsize,
                "func.fpd",                    func.fpd,
                "ida_frame.frame_off_retaddr", ida_frame.frame_off_retaddr(func),
                "ida_frame.frame_off_lvars",   ida_frame.frame_off_lvars(func),
                "ida_frame.frame_off_args",    ida_frame.frame_off_args(func),
                "ida_frame.frame_off_savregs", ida_frame.frame_off_savregs(func),
                ))

    stkvars = indexBy(dump_stkvars(ea), 'name')
    lvars = []
    if vu and func:
        stk_lvars = [(n.name, n.tif.get_size(), 
            n.location.stkoff(),
            ) for n in vu.cfunc.lvars if n.location.is_stkoff()]

        for name, size, offset in stk_lvars:
            o = AttrDict()
            o.update({
                'name': name,
                'size': size,
                'lvar_offset': offset,
            })
            lvars.append(o)
    lvars = indexBy(lvars, 'name')

    lvar_names = lvars.keys()
    for name in lvar_names:
        if name in stkvars:
            print({
                'name': name,
                'lvar_offset': lvars[name].lvar_offset,
                'stk_offset': stkvars[name].offset,
            })

class AttrDict(dict):
    def __init__(self, *args, **kwargs):
        super(AttrDict, self).__init__(*args, **kwargs)
        self.__dict__ = self

Answer 1

I just found what appears to be the official version of the adjustment function for the transmigration of worthless min-spd based offsets into heaven sent frame based offsets.

cfunc.get_stkoff_delta()

Possible implementation:

def GetMinSpdAdjustment(funcea):
    func = ida_funcs.get_func(funcea)
    return 0 - (func.frsize + func.frregs + idc.get_spd(idc.get_min_spd_ea(func.start_ea)))

Example usage:

vu = idaapi.open_pseudocode(funcea, 0)
vu.cfunc.lvars[10].get_stkoff() - vu.cfunc.get_stkoff_delta()

Complete example:

# call rename_lvar(old, new, ea)

def get_pseudocode_vu(ea, vu):
    if vu:
        return vu
    
    return idaapi.open_pseudocode(ea, 0)

def label_stkvar(offset, name, ea=None, vu=None):
    sid = idc.get_frame_id(ea)
    old_name = idc.get_member_name(sid, offset)
    if old_name:
        if old_name == name:
            return old_name

        if idc.set_member_name(sid, offset, name):
            return old_name

def rename_lvar(old, new, ea, uniq=0, vu=None):
    def make_unique_name(name, taken):
        if name not in taken:
            return name
        fmt = "%s_%%i" % name
        for i in range(3, 1<<10):
            name = fmt % i
            if name not in taken:
                return name

    old = old.strip()
    new = new.strip()
    if old == new:
        return True

    vu = get_pseudocode_vu(ea, vu)
    names = [n.name for n in vu.cfunc.lvars]

    if new in names:
        if uniq:
            return False

        new = make_unique_name(new, names)

    lvars = [n for n in vu.cfunc.lvars if n.name == old]
    if lvars:
        lvar = lvars[0]
        if lvar.is_stk_var():
            offset = lvar.get_stkoff() - vu.cfunc.get_stkoff_delta()
            old_name = label_stkvar(offset, new, ea=ea, vu=vu)

        return vu.rename_lvar(lvar, new, 1)

Answer 2

Based on the information we discuss in the comments, I have an idea about supporting such cases. You can use idc.get_frame_regs_size(ea) to adjust the offset. Thus can do something like:

adjust_offset = idc.get_frame_regs_size(ea)
[n.get_stkoff()+-adjust_offset for n in vu.cfunc.lvars if n.is_stk_var()]

according to your implementation.

You might as well want to consider other endpoints like -

print(idc.get_frame_lvar_size(ea))
print(idc.get_frame_args_size(ea))
print(idc.get_frame_size(ea))

to experiment more about adjusting the offset.

I am sure there has to be a robust solution, and this may not be perfect (I believe this should work in theory). But, give it a try.