Crash, the Linux kernel core dump debugger, has the ability to use extensions written in C, but oftentimes we want to do some automatic analysis or a dump of data that is more involved than what Crash offers natively. Since GDB 7.0, GDB has incorporated an embedded python interpreter.

Since we started hacking on this project during Hack Week several years ago, we've made significant progress in making crash-python more usable. The initial plan of exporting crash functionality via Python was ultimately discarded in favor of writing a kernel debugger in Python with as few GDB extensions as possible. The GDB extensions have long since been written and are entirely generic. We export some ELF functionality as well as add the ability to implement a GDB Target in Python.

As of the last Hack Week, the target has been implemented and interfaces with Petr Tesarik's libkdumpfile python module. The crash-python project uses that to offer some basic kernel debugging functionality: We can do 'dmesg', 'sys', and subsets of 'ps' and 'kmem.' We can also load up the tasks from the crash dump as GDB threads and do full symbolic debugging (with the caveat that the kernel is built with -O2 and some symbols are optimized away.) Some colleagues have used it to hunt real bugs in the course of their normal work!

This Hack Week, I'd like to continue to implement the full functionality of the original crash tool in Python with the end goal of being able to replace crash itself.

Latest links: <li><a href="https://build.opensuse.org/project/show/home![add-emoji](https://assets-cdn.github.com/images/icons/emoji/jeff_mahoney.png)crash-python">crash-python OBS Project</a> - Contains packages for libkdumpfile and our extended GDB without having to build them manually. <li><a href="https://github.com/jeffmahoney/crash-python/tree/crash-wip">crash-python GitHub Project (WIP branch)</a> - Contains the latest Python code <li><a href="https://etherpad.nue.suse.com/p/crash-todo">Etherpad for Crash TODO list</a>

Looking for mad skills in:

Nothing? Add some keywords!

This project is part of:

Hack Week 11 Hack Week 12 Hack Week 13 Hack Week 14 Hack Week 15 Hack Week 16 Hack Week 17

Activity

  • 11 months ago: dsterba liked crash-python
  • 11 months ago: mbrugger liked crash-python
  • 11 months ago: wanghaisu liked crash-python
  • 11 months ago: mwilck liked crash-python
  • 11 months ago: yosun liked crash-python
  • Show History

    Comments

    • ptesarik
      almost 4 years ago by ptesarik | Reply

      The main difficulty is that crash translates everything to PHYSICAL addresses, whereas gdb uses VIRTUAL addresses only. Kernel crash dumps generally do not contain all existing virtual mappings of physical RAM, so gdb does not find the data. I'm not sure how difficult it would be to implement a kind of virttophys() in gdb, but we're pretty much lost without that.

    • ptesarik
      almost 4 years ago by ptesarik | Reply

      To make things clear, this project's goals are very different from those of the existing Python/CRASH API at http://sourceforge.net/projects/pykdump/

    • jeff_mahoney
      almost 4 years ago by jeff_mahoney | Reply

      Useful links:

      https://build.opensuse.org/project/show/homeadd-emojipy-crash https://etherpad.nue.suse.com/p/crash-todo https://github.com/jeffmahoney/py-crash

    • jeff_mahoney
      almost 4 years ago by jeff_mahoney | Reply

      @Petr For the python code I've already implemented (see github link), it does the translation already.

      • ptesarik
        almost 4 years ago by ptesarik | Reply

        Yes, but most of it is missing. It does not handle relocated kernels, vmemmap, kernel modules and a lot of other stuff.

        • jeff_mahoney
          almost 4 years ago by jeff_mahoney | Reply

          Yep. It wasn't intended to be complete yet. It's more of a starting point.

      • ptesarik
        almost 4 years ago by ptesarik | Reply

        And most importantly, the interface to gdb still uses virtual addresses. Note that the dump file may not contain the identity-mapped page of all physical pages; and some architectures do not even map all physical pages to linear address space (yeah, I know that PAE sucks, but...).

    Similar Projects

    This project is one of its kind!