What are some useful strategies for tackling a list of undocumented assembly language instructions? Even as someone who writes assembly code, I find it to be challenging at times to understand someone else's code because the nature of assembly language means mentally having to keep track of CPU state at various points during the program. I've seen some success with getting a notepad and a pen and working through the program by hand. What are some other strategies? Let's take for example this function:
; cmp edx,10
; jae internal_error
push ebx ecx
push eax
mov ebx,eax
mov ecx,edx
shld edx,eax,2
sub ebx,edx
sbb ecx,0
mov eax,ebx
mov ebx,1999999Ah
mul ebx
mov eax,ecx
imul eax,ebx
add eax,edx
pop edx
imul ecx,eax,10
sub edx,ecx
cmp edx,10
jb somewhere
sub edx,10
inc eax
pop ecx ebx
retn
Due to the nature of assembly language, it is not immediately apparent "what this code actually does" at a high level. To me, the process of understanding this is true "reverse engineering" but it doesn't necessarily need to apply to reading from a disassembler, it could just be someone else's assembler source code as well. In fact, sometimes without the help of IDA Pro or Binary Ninja disassembling a binary written in a HLL, reverse engineering assembly written in aassembly is more difficult.