I initially thought you wouldn't be able to dump the program, but it turns out that you can — see the second section of this answer.
Running the program
Most of the usual methods won't work because the executable is setuid. If you start the program normally, it runs with elevated privileges (euid ≠ ruid), and most debugging facilities are reserved to root. For example, anything that relies on attaching to the program with
ptrace is reserved to root. This includes just about anything you could do in a debugger. Linux lets you dump the memory of a running process (with
ptrace or through
/proc/$pid/mem), but that too is disabled for a setuid executable.
You can run
strace ./bin, and see what system calls the program makes. However, this runs the program without any extra privileges. It may well stop early and complain that it can't read a file, or that it isn't running as the right user.
You can see statistics like CPU usage, IO usage, network usage, memory usage, etc. in
/proc/$pid: files like
net/netstat are world-readable (whether the program is actually started with extra privileges or not, the setuid restrictions apply). The one interesting thing I see is the program's network connections. The really juicy stuff, like memory contents (
mem) and even open files (
fd) is disabled either way. Preloading a library with
LD_PRELOAD or running the program with a different
LD_LIBRARY_PATH is also disabled either way. Nor will you get a core dump.
So what works? You can see open network connections, maybe one of them will give you a clue or will be spoofable. You can try to figure out what files the program accesses by modifying them when you can, or by running the program from another directory (try making a symbolic link).
If you have access to a chroot jail on the machine, try running the program from there. If you can run a virtual machine which implements shared folders (e.g. VMware or VirtualBox), see if you can access the program from there. This is all about subverting the security of the system: the permissions are designed to hinder you in your task.
An advanced technique is to observe the program under heavy load and try to deduce what it does from the time it takes. Pollute the disk caches and see what files become quickly accessible. See how the swap usage varies depending on what the program does (so you get an idea of how many different pages are in its working set, not just how much total memory it's using). A lot of trial and error will be required. When you know the code and are trying to obtain the data, this is a well-known side channel attack. When you don't know the code, I expect it to be very difficult to obtain useful information this way.
Dumping the memory contents
I initially thought you couldn't dump the memory of the program because of the setuid bit (which usually wouldn't be a problem because you'd just copy the executable, but here this isn't possible because the executable is not readable). This turns out to be wrong. If you attach to the program with
ptrace before calling
execve (like running
strace ./bin does), you do have access to
ptrace in all its glory, including
PTRACE_PEEKEXEC. That doesn't run the program setuid, so it may behave differently, but the code and static data loaded into memory are the same either way.
Kudos to samuirai for pointing this out.
Here's an overview of how you can dump the program's memory:
- Fork, and call
ptrace(PTRACE_TRACEME, ...) in the child then execute the program (
execl("./bin", "./bin", NULL)).
- In the parent, call
waitpid to wait for the child to return from
- Figure out the address at which the code is loaded. The value of the IP register (obtained through
PTRACE_GETREGS) should give a clue, and you can look for file format headers.
- Read the child's memory word by word (
- Reconstruct the file format (e.g. make an ELF binary from text and data sections) if needed. Dump out the data into a file.
- Kill the child — it won't even get to execute anything.
Dougall pointed out a working proof-of-concept tool for ELF executables: xocopy.