I am looking for Dynamic Symbolic Execution tools
Probably nitpicking, but as your question is about PIN, this tool falls under the DBI (Dynamic Binary Instrumentation) tool category, it has nothing to do with symbolic execution.
Does it work with a symbolic CPU ?
No, you need a binary to be instrumented, be it a PE (Windows) or ELF (linux).
Or is it a sort of debugger which interacts with kernel and real process ?
PIN has two modes of operation:
JIT mode: the code from the binary (that is being instrumented) is JITed on the fly by the PIN engine, which allows you to put your "hooks" everywhere you need to.
Probe mode: the code is executed "as is" but you can still put some hooks only at some places (using "trampolines" to redirect the code from the binary to your own places). This is a more limited mode compared to the JIT mode.
Note that PIN is user-mode only (it doesn't deal with the kernel side of the operating system).
If so, i think there is a way to detect pintool trace for a malware ?
It's pretty easy to detect that your application is currently instrumented by PIN. It has never meant to be undetectable. The pin engine inject your pintool (where you code your instrumentation and analysis routines) into the process address space and either JIT the assembly code on the fly or puts hooks / trampolines into the code, so it's easily detectable.
The simplest way to detect PIN being just checking in the process module list if the pin engine is loaded.
If you want to detect PIN, just check this presentation from Recon: Dynamic Binary Instrumentation Frameworks: I know you're there spying on me.
