Debugging a position independent executable ELF

Question

I am trying to follow the write-up about hackercalc (a challenge for DEFCON 2015 quals). First, by loading the challenge into gdb, I observe that:

(gdb) info files
...
Entry point: 0xcf8
...

So I set a breakpoint at the entry point and run the program:

(gdb) break *0xcf8
Breakpoint 1 at 0xcf8
(gdb) run
Starting program: /home/tathanhdinh/CTFs/hackercalc
Error in re-setting breakpoint 1: Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0xcf8

Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0xcf8

That seems this breakpoint does not work. I do not understand why it says Cannot access memory at address 0xcf8 but I guess that there was some code which runs before has made some interference (so the program cannot reach this entry point), as far as I know, this code must be the elf interpreter of the program.

So, I look into segments of the program to check this interpreter, all seems to be normal (except that readelf says the program is a shared object file, but I think that is not important):

readelf --segments hackercalc

Elf file type is DYN (Shared object file)
Entry point 0xcf8
There are 8 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  PHDR           0x000034 0x00000034 0x00000034 0x00100 0x00100 R E 0x4
  INTERP         0x000154 0x00000154 0x00000154 0x00013 0x00013 R   0x1
      [Requesting program interpreter: /lib/ld-linux.so.2]
  LOAD           0x000000 0x00000000 0x00000000 0x16b4a 0x16b4a R E 0x1000
  LOAD           0x016eac 0x00017eac 0x00017eac 0x001e4 0x001ec RW  0x1000

Now, I try to get again information about this process:

(gdb) info files
Symbols from "/home/tathanhdinh/CTFs/hackercalc".
Native process:
        Using the running image of child process 10545.
        While running this, GDB does not access memory from...
Local exec file:
        `/home/tathanhdinh/CTFs/hackercalc', file type elf32-i386.
        Entry point: 0x56555cf8

And I am totally lost here, why the elf interpreter (which is ld-linux.so.2) can create a new process in this case?

I try another approach by loading the challenge into IDA, set a breakpoint at the entry point, then run but IDA simply hangs up!!!

So my question is: how can this happen? Am I totally wrong about the code executed before the entry point? Does the challenge use some special tricks to bypass debugging?

NB1. Thanks to @Igor, I finally find out how to debug this challenge. In summary, I disable ALSR to get a stable address of the entry point. Then I load it into gdbserver, use IDA to attach this debug session, set a breakpoint at the entry point (whose address is stable by disabling ALSR).