I'm dealing with a custom-upx packed library that I'm trying to unpack. System is macOS.
Given it's a library/plugin, it doesn't have a standalone entrypoint. I ran it through a disassembler but wasn't able to find obvious hints where the OEP might be. On top of that, it also uses anti-debugging measurements through ptrace to make it even harder.
Next I wrote my own loader for it including callbacks it wants, and managed to get it to load, then used lldb save-core to dump the entire memory (of my loader+lib) into a 3GB-ish file.
Here is where I am stuck: Memory analysis and forensics doesn't look like its popular on mac with most tools outdated and no longer working. I don't know how to approach this going forward and if it's even possible to restore the unpacked variant from this dump.