BUG-HUNTING 8.1 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247
  1. Table of contents
  2. =================
  3. Last updated: 20 December 2005
  4. Contents
  5. ========
  6. - Introduction
  7. - Devices not appearing
  8. - Finding patch that caused a bug
  9. -- Finding using git-bisect
  10. -- Finding it the old way
  11. - Fixing the bug
  12. Introduction
  13. ============
  14. Always try the latest kernel from kernel.org and build from source. If you are
  15. not confident in doing that please report the bug to your distribution vendor
  16. instead of to a kernel developer.
  17. Finding bugs is not always easy. Have a go though. If you can't find it don't
  18. give up. Report as much as you have found to the relevant maintainer. See
  19. MAINTAINERS for who that is for the subsystem you have worked on.
  20. Before you submit a bug report read REPORTING-BUGS.
  21. Devices not appearing
  22. =====================
  23. Often this is caused by udev. Check that first before blaming it on the
  24. kernel.
  25. Finding patch that caused a bug
  26. ===============================
  27. Finding using git-bisect
  28. ------------------------
  29. Using the provided tools with git makes finding bugs easy provided the bug is
  30. reproducible.
  31. Steps to do it:
  32. - start using git for the kernel source
  33. - read the man page for git-bisect
  34. - have fun
  35. Finding it the old way
  36. ----------------------
  37. [Sat Mar 2 10:32:33 PST 1996 KERNEL_BUG-HOWTO lm@sgi.com (Larry McVoy)]
  38. This is how to track down a bug if you know nothing about kernel hacking.
  39. It's a brute force approach but it works pretty well.
  40. You need:
  41. . A reproducible bug - it has to happen predictably (sorry)
  42. . All the kernel tar files from a revision that worked to the
  43. revision that doesn't
  44. You will then do:
  45. . Rebuild a revision that you believe works, install, and verify that.
  46. . Do a binary search over the kernels to figure out which one
  47. introduced the bug. I.e., suppose 1.3.28 didn't have the bug, but
  48. you know that 1.3.69 does. Pick a kernel in the middle and build
  49. that, like 1.3.50. Build & test; if it works, pick the mid point
  50. between .50 and .69, else the mid point between .28 and .50.
  51. . You'll narrow it down to the kernel that introduced the bug. You
  52. can probably do better than this but it gets tricky.
  53. . Narrow it down to a subdirectory
  54. - Copy kernel that works into "test". Let's say that 3.62 works,
  55. but 3.63 doesn't. So you diff -r those two kernels and come
  56. up with a list of directories that changed. For each of those
  57. directories:
  58. Copy the non-working directory next to the working directory
  59. as "dir.63".
  60. One directory at time, try moving the working directory to
  61. "dir.62" and mv dir.63 dir"time, try
  62. mv dir dir.62
  63. mv dir.63 dir
  64. find dir -name '*.[oa]' -print | xargs rm -f
  65. And then rebuild and retest. Assuming that all related
  66. changes were contained in the sub directory, this should
  67. isolate the change to a directory.
  68. Problems: changes in header files may have occurred; I've
  69. found in my case that they were self explanatory - you may
  70. or may not want to give up when that happens.
  71. . Narrow it down to a file
  72. - You can apply the same technique to each file in the directory,
  73. hoping that the changes in that file are self contained.
  74. . Narrow it down to a routine
  75. - You can take the old file and the new file and manually create
  76. a merged file that has
  77. #ifdef VER62
  78. routine()
  79. {
  80. ...
  81. }
  82. #else
  83. routine()
  84. {
  85. ...
  86. }
  87. #endif
  88. And then walk through that file, one routine at a time and
  89. prefix it with
  90. #define VER62
  91. /* both routines here */
  92. #undef VER62
  93. Then recompile, retest, move the ifdefs until you find the one
  94. that makes the difference.
  95. Finally, you take all the info that you have, kernel revisions, bug
  96. description, the extent to which you have narrowed it down, and pass
  97. that off to whomever you believe is the maintainer of that section.
  98. A post to linux.dev.kernel isn't such a bad idea if you've done some
  99. work to narrow it down.
  100. If you get it down to a routine, you'll probably get a fix in 24 hours.
  101. My apologies to Linus and the other kernel hackers for describing this
  102. brute force approach, it's hardly what a kernel hacker would do. However,
  103. it does work and it lets non-hackers help fix bugs. And it is cool
  104. because Linux snapshots will let you do this - something that you can't
  105. do with vendor supplied releases.
  106. Fixing the bug
  107. ==============
  108. Nobody is going to tell you how to fix bugs. Seriously. You need to work it
  109. out. But below are some hints on how to use the tools.
  110. To debug a kernel, use objdump and look for the hex offset from the crash
  111. output to find the valid line of code/assembler. Without debug symbols, you
  112. will see the assembler code for the routine shown, but if your kernel has
  113. debug symbols the C code will also be available. (Debug symbols can be enabled
  114. in the kernel hacking menu of the menu configuration.) For example:
  115. objdump -r -S -l --disassemble net/dccp/ipv4.o
  116. NB.: you need to be at the top level of the kernel tree for this to pick up
  117. your C files.
  118. If you don't have access to the code you can also debug on some crash dumps
  119. e.g. crash dump output as shown by Dave Miller.
  120. > EIP is at ip_queue_xmit+0x14/0x4c0
  121. > ...
  122. > Code: 44 24 04 e8 6f 05 00 00 e9 e8 fe ff ff 8d 76 00 8d bc 27 00 00
  123. > 00 00 55 57 56 53 81 ec bc 00 00 00 8b ac 24 d0 00 00 00 8b 5d 08
  124. > <8b> 83 3c 01 00 00 89 44 24 14 8b 45 28 85 c0 89 44 24 18 0f 85
  125. >
  126. > Put the bytes into a "foo.s" file like this:
  127. >
  128. > .text
  129. > .globl foo
  130. > foo:
  131. > .byte .... /* bytes from Code: part of OOPS dump */
  132. >
  133. > Compile it with "gcc -c -o foo.o foo.s" then look at the output of
  134. > "objdump --disassemble foo.o".
  135. >
  136. > Output:
  137. >
  138. > ip_queue_xmit:
  139. > push %ebp
  140. > push %edi
  141. > push %esi
  142. > push %ebx
  143. > sub $0xbc, %esp
  144. > mov 0xd0(%esp), %ebp ! %ebp = arg0 (skb)
  145. > mov 0x8(%ebp), %ebx ! %ebx = skb->sk
  146. > mov 0x13c(%ebx), %eax ! %eax = inet_sk(sk)->opt
  147. In addition, you can use GDB to figure out the exact file and line
  148. number of the OOPS from the vmlinux file. If you have
  149. CONFIG_DEBUG_INFO enabled, you can simply copy the EIP value from the
  150. OOPS:
  151. EIP: 0060:[<c021e50e>] Not tainted VLI
  152. And use GDB to translate that to human-readable form:
  153. gdb vmlinux
  154. (gdb) l *0xc021e50e
  155. If you don't have CONFIG_DEBUG_INFO enabled, you use the function
  156. offset from the OOPS:
  157. EIP is at vt_ioctl+0xda8/0x1482
  158. And recompile the kernel with CONFIG_DEBUG_INFO enabled:
  159. make vmlinux
  160. gdb vmlinux
  161. (gdb) p vt_ioctl
  162. (gdb) l *(0x<address of vt_ioctl> + 0xda8)
  163. or, as one command
  164. (gdb) l *(vt_ioctl + 0xda8)
  165. If you have a call trace, such as :-
  166. >Call Trace:
  167. > [<ffffffff8802c8e9>] :jbd:log_wait_commit+0xa3/0xf5
  168. > [<ffffffff810482d9>] autoremove_wake_function+0x0/0x2e
  169. > [<ffffffff8802770b>] :jbd:journal_stop+0x1be/0x1ee
  170. > ...
  171. this shows the problem in the :jbd: module. You can load that module in gdb
  172. and list the relevant code.
  173. gdb fs/jbd/jbd.ko
  174. (gdb) p log_wait_commit
  175. (gdb) l *(0x<address> + 0xa3)
  176. or
  177. (gdb) l *(log_wait_commit + 0xa3)
  178. Another very useful option of the Kernel Hacking section in menuconfig is
  179. Debug memory allocations. This will help you see whether data has been
  180. initialised and not set before use etc. To see the values that get assigned
  181. with this look at mm/slab.c and search for POISON_INUSE. When using this an
  182. Oops will often show the poisoned data instead of zero which is the default.
  183. Once you have worked out a fix please submit it upstream. After all open
  184. source is about sharing what you do and don't you want to be recognised for
  185. your genius?
  186. Please do read Documentation/SubmittingPatches though to help your code get
  187. accepted.