1

I wanted to debug the hanging ruby process, but instead my kernel crashes each time when I'm doing the following steps:

  1. I'm re-running my ruby script (vagrant) which hangs:

    $ vagrant destroy -f && VAGRANT_LOG=info vagrant up
    ...
    INFO ssh: Setting SSH_AUTH_SOCK remotely: /tmp/ssh-FP9viQfD9J/agent.1425
    (process hanging)
    load: 2.48  cmd: ruby 1384 waiting 2.90u 0.62s
    
  2. I'm attaching lldb into the running process (on different terminal):

    $ lldb -p $(pgrep -fn ruby)
    

    Popup: Developer Tools Access needs to take control of another process for debugging to continue. Type your password. Continue.

    (lldb) process attach --pid 1442
    Process 1442 stopped
    Executable module set to "/opt/vagrant/bin/../embedded/bin/ruby".
    Architecture set to: x86_64-apple-macosx.
    
    (lldb) thread list
    Process 1442 stopped
    * thread #1: tid = 0x2427, 0x00007fff84f3f716 libsystem_kernel.dylib`__psynch_cvwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
      thread #2: tid = 0x245d, 0x00007fff84f3f9aa libsystem_kernel.dylib`__select + 10
      thread #3: tid = 0x2a8d, 0x00007fff84f4094a libsystem_kernel.dylib`poll + 10
      thread #4: tid = 0x2a9a, 0x00007fff84f3f716 libsystem_kernel.dylib`__psynch_cvwait + 10
    
  3. I'm calling rb_backtrace() for thread #1:

    (lldb) call (void)rb_backtrace()
    

    which prints for me the ruby backtrace as expected (on process foreground):

    from /opt/vagrant/embedded/gems/gems/vagrant-1.7.2/plugins/communicators/ssh/communicator.rb:566:in `block in shell_execute'
    from /opt/vagrant/embedded/gems/gems/vagrant-1.7.2/plugins/communicators/ssh/communicator.rb:566:in `loop'
    
  4. Since ruby is using different threads, I wanted to do the same for thread #2 (which failed):

    (lldb) thread select 2
    * thread #2: tid = 0x245d, 0x00007fff84f3f9aa libsystem_kernel.dylib`__select + 10
        frame #0: 0x00007fff84f3f9aa libsystem_kernel.dylib`__select + 10
    libsystem_kernel.dylib`__select + 10:
    -> 0x7fff84f3f9aa:  jae    0x7fff84f3f9b4            ; __select + 20
       0x7fff84f3f9ac:  movq   %rax, %rdi
       0x7fff84f3f9af:  jmp    0x7fff84f3c19a            ; cerror
       0x7fff84f3f9b4:  retq   
    
    (lldb) thread list
    Process 1442 stopped
      thread #1: tid = 0x2427, 0x00007fff84f3f716 libsystem_kernel.dylib`__psynch_cvwait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    * thread #2: tid = 0x245d, 0x00007fff84f3f9aa libsystem_kernel.dylib`__select + 10
      thread #3: tid = 0x2a8d, 0x00007fff84f4094a libsystem_kernel.dylib`poll + 10
      thread #4: tid = 0x2a9a, 0x00007fff84f3f716 libsystem_kernel.dylib`__psynch_cvwait + 10
    
    (lldb) call (void)rb_backtrace()
    

    Kernel panic screen: You need to restart your computer.

  5. At this point the kernel crashed with the following backtrace found in log:

    panic(cpu 2 caller 0xffffff80022dcc1d): Kernel trap at 0xffffff80025f69e4, type 14=page fault, registers:
    Fault CR2: 0x0000000001f6d695, Error code: 0x0000000000000000, Fault CPU: 0x2
    Backtrace (CPU 2), Frame : Return Address
    0xffffff81f0ca3aa0 : 0xffffff8002223139 
    0xffffff81f0ca3b20 : 0xffffff80022dcc1d 
    ...
    BSD process name corresponding to current thread: ruby
    
    Mac OS version:
    13F1077
    
    Kernel version:
    Darwin Kernel Version 13.4.0: Wed Mar 18 16:20:14 PDT 2015; root:xnu-2422.115.14~1/RELEASE_X86_64
    last loaded kext at 244023643290: com.apple.filesystems.msdosfs 1.9 (addr 0xffffff7f8308c000, size 65536)
    loaded kexts:
    foo.tap 1.0
    foo.tun 1.0
    org.virtualbox.kext.VBoxNetAdp  4.3.12
    org.virtualbox.kext.VBoxNetFlt  4.3.12
    org.virtualbox.kext.VBoxUSB 4.3.12
    org.virtualbox.kext.VBoxDrv 4.3.12
    com.apple.filesystems.msdosfs   1.9
    ...
    

This happened already twice.

What was the reason for a kernel crash? Is it a lldb/kernel bug, or expected behaviour by doing something wrong? If so, what should be the correct safer approach of calling rb_backtrace() on different threads (without crashing the kernel)?

For clarification, ruby script (vagrant) was run without sudo, the same as lldb.

I'm using lldb-320.4.160.

  • If you are running kdumpd on your main machine, then you can catch the panic dumps that way if you are but you also need to set the nvram properly as well. So on the remote machine you need to have setup ahead of time in the nvram: boot-args=-v debug=0xd44 _panicd_ip=127.55.56.29 replace 127.55.56.59 with your IP address. See Apple document describing how to do it here. Is there a reason why you can't just run lldb directly on your debug machine? If you could you could just do a bt directly on the other thread. If – Div Sep 10 '16 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.