32 and 64 differ in another way than you think. You have pretty much the same binary code but on 64 you operate on 64-bit registers and 64-bit data. 32 and 64 are about data not code. Using OllyDBG on 64-bit code is ok, but:
you have to take attention that you operating on 64-bit data.
ie. when Olly shows MOV DWORD[PTR], EAX then really moves 8 bytes.