Skip to content

Commit

Permalink
s390/docs: Remove sections that are not related to s390
Browse files Browse the repository at this point in the history 
Information how to use the GCC pre-processor, objdump, strace, top, etc.
are generic and not specific to the S390 architecture, so we do not need
this information in Debugging390.txt

Signed-off-by: Thomas Huth <[email protected]>
Signed-off-by: Martin Schwidefsky <[email protected]>
  • Loading branch information
@huth
huth authored and Martin Schwidefsky committed Nov 18, 2014
1 parent b195562 commit a6b42af
Showing 1 changed file with 0 additions and 374 deletions.
374 changes: 0 additions & 374 deletions Documentation/s390/Debugging390.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,6 @@ The Linux for s/390 & z/Architecture Kernel Task Structure
Register Usage & Stackframes on Linux for s/390 & z/Architecture
A sample program with comments
Compiling programs for debugging on Linux for s/390 & z/Architecture
Figuring out gcc compile errors
Debugging Tools
objdump
strace
Performance Debugging
Debugging under VM
s/390 & z/Architecture IO Overview
Debugging IO on s/390 & z/Architecture under VM
Expand Down Expand Up @@ -740,376 +735,7 @@ Debugging with optimisation has since much improved after fixing
some bugs, please make sure you are using gdb-5.0 or later developed
after Nov'2000.

Figuring out gcc compile errors
===============================
If you are getting a lot of syntax errors compiling a program & the problem
isn't blatantly obvious from the source.
It often helps to just preprocess the file, this is done with the -E
option in gcc.
What this does is that it runs through the very first phase of compilation
( compilation in gcc is done in several stages & gcc calls many programs to
achieve its end result ) with the -E option gcc just calls the gcc preprocessor (cpp).
The c preprocessor does the following, it joins all the files #included together
recursively ( #include files can #include other files ) & also the c file you wish to compile.
It puts a fully qualified path of the #included files in a comment & it
does macro expansion.
This is useful for debugging because
1) You can double check whether the files you expect to be included are the ones
that are being included ( e.g. double check that you aren't going to the i386 asm directory ).
2) Check that macro definitions aren't clashing with typedefs,
3) Check that definitions aren't being used before they are being included.
4) Helps put the line emitting the error under the microscope if it contains macros.

For convenience the Linux kernel's makefile will do preprocessing automatically for you
by suffixing the file you want built with .i ( instead of .o )

e.g.
from the linux directory type
make arch/s390/kernel/signal.i
this will build

s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -E arch/s390/kernel/signal.c
> arch/s390/kernel/signal.i

Now look at signal.i you should see something like.


# 1 "/home1/barrow/linux/include/asm/types.h" 1
typedef unsigned short umode_t;
typedef __signed__ char __s8;
typedef unsigned char __u8;
typedef __signed__ short __s16;
typedef unsigned short __u16;

If instead you are getting errors further down e.g.
unknown instruction:2515 "move.l" or better still unknown instruction:2515
"Fixme not implemented yet, call Martin" you are probably are attempting to compile some code
meant for another architecture or code that is simply not implemented, with a fixme statement
stuck into the inline assembly code so that the author of the file now knows he has work to do.
To look at the assembly emitted by gcc just before it is about to call gas ( the gnu assembler )
use the -S option.
Again for your convenience the Linux kernel's Makefile will hold your hand &
do all this donkey work for you also by building the file with the .s suffix.
e.g.
from the Linux directory type
make arch/s390/kernel/signal.s

s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer
-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -S arch/s390/kernel/signal.c
-o arch/s390/kernel/signal.s


This will output something like, ( please note the constant pool & the useful comments
in the prologue to give you a hand at interpreting it ).

.LC54:
.string "misaligned (__u16 *) in __xchg\n"
.LC57:
.string "misaligned (__u32 *) in __xchg\n"
.L$PG1: # Pool sys_sigsuspend
.LC192:
.long -262401
.LC193:
.long -1
.LC194:
.long schedule-.L$PG1
.LC195:
.long do_signal-.L$PG1
.align 4
.globl sys_sigsuspend
.type sys_sigsuspend,@function
sys_sigsuspend:
# leaf function 0
# automatics 16
# outgoing args 0
# need frame pointer 0
# call alloca 0
# has varargs 0
# incoming args (stack) 0
# function length 168
STM 8,15,32(15)
LR 0,15
AHI 15,-112
BASR 13,0
.L$CO1: AHI 13,.L$PG1-.L$CO1
ST 0,0(15)
LR 8,2
N 5,.LC192-.L$PG1(13)

Adding -g to the above output makes the output even more useful
e.g. typing
make CC:="s390-gcc -g" kernel/sched.s

which compiles.
s390-gcc -g -D__KERNEL__ -I/home/barrow/linux-2.3/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strict-aliasing -pipe -fno-strength-reduce -S kernel/sched.c -o kernel/sched.s

also outputs stabs ( debugger ) info, from this info you can find out the
offsets & sizes of various elements in structures.
e.g. the stab for the structure
struct rlimit {
unsigned long rlim_cur;
unsigned long rlim_max;
};
is
.stabs "rlimit:T(151,2)=s8rlim_cur:(0,5),0,32;rlim_max:(0,5),32,32;;",128,0,0,0
from this stab you can see that
rlimit_cur starts at bit offset 0 & is 32 bits in size
rlimit_max starts at bit offset 32 & is 32 bits in size.


Debugging Tools:
================

objdump
=======
This is a tool with many options the most useful being ( if compiled with -g).
objdump --source <victim program or object file> > <victims debug listing >


The whole kernel can be compiled like this ( Doing this will make a 17MB kernel
& a 200 MB listing ) however you have to strip it before building the image
using the strip command to make it a more reasonable size to boot it.

A source/assembly mixed dump of the kernel can be done with the line
objdump --source vmlinux > vmlinux.lst
Also, if the file isn't compiled -g, this will output as much debugging information
as it can (e.g. function names). This is very slow as it spends lots
of time searching for debugging info. The following self explanatory line should be used
instead if the code isn't compiled -g, as it is much faster:
objdump --disassemble-all --syms vmlinux > vmlinux.lst

As hard drive space is valuable most of us use the following approach.
1) Look at the emitted psw on the console to find the crash address in the kernel.
2) Look at the file System.map ( in the linux directory ) produced when building
the kernel to find the closest address less than the current PSW to find the
offending function.
3) use grep or similar to search the source tree looking for the source file
with this function if you don't know where it is.
4) rebuild this object file with -g on, as an example suppose the file was
( /arch/s390/kernel/signal.o )
5) Assuming the file with the erroneous function is signal.c Move to the base of the
Linux source tree.
6) rm /arch/s390/kernel/signal.o
7) make /arch/s390/kernel/signal.o
8) watch the gcc command line emitted
9) type it in again or alternatively cut & paste it on the console adding the -g option.
10) objdump --source arch/s390/kernel/signal.o > signal.lst
This will output the source & the assembly intermixed, as the snippet below shows
This will unfortunately output addresses which aren't the same
as the kernel ones you should be able to get around the mental arithmetic
by playing with the --adjust-vma parameter to objdump.




static inline void spin_lock(spinlock_t *lp)
{
a0: 18 34 lr %r3,%r4
a2: a7 3a 03 bc ahi %r3,956
__asm__ __volatile(" lhi 1,-1\n"
a6: a7 18 ff ff lhi %r1,-1
aa: 1f 00 slr %r0,%r0
ac: ba 01 30 00 cs %r0,%r1,0(%r3)
b0: a7 44 ff fd jm aa <sys_sigsuspend+0x2e>
saveset = current->blocked;
b4: d2 07 f0 68 mvc 104(8,%r15),972(%r4)
b8: 43 cc
return (set->sig[0] & mask) != 0;
}

6) If debugging under VM go down to that section in the document for more info.


I now have a tool which takes the pain out of --adjust-vma
& you are able to do something like
make /arch/s390/kernel/traps.lst
& it automatically generates the correctly relocated entries for
the text segment in traps.lst.
This tool is now standard in linux distro's in scripts/makelst

strace:
-------
Q. What is it ?
A. It is a tool for intercepting calls to the kernel & logging them
to a file & on the screen.

Q. What use is it ?
A. You can use it to find out what files a particular program opens.



Example 1
---------
If you wanted to know does ping work but didn't have the source
strace ping -c 1 127.0.0.1
& then look at the man pages for each of the syscalls below,
( In fact this is sometimes easier than looking at some spaghetti
source which conditionally compiles for several architectures ).
Not everything that it throws out needs to make sense immediately.

Just looking quickly you can see that it is making up a RAW socket
for the ICMP protocol.
Doing an alarm(10) for a 10 second timeout
& doing a gettimeofday call before & after each read to see
how long the replies took, & writing some text to stdout so the user
has an idea what is going on.

socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
getuid() = 0
setuid(0) = 0
stat("/usr/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory)
stat("/usr/share/locale/libc/C", 0xbffff134) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory)
getpid() = 353
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [49152], 4) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(3, 1), ...}) = 0
mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40008000
ioctl(1, TCGETS, {B9600 opost isig icanon echo ...}) = 0
write(1, "PING 127.0.0.1 (127.0.0.1): 56 d"..., 42PING 127.0.0.1 (127.0.0.1): 56 data bytes
) = 42
sigaction(SIGINT, {0x8049ba0, [], SA_RESTART}, {SIG_DFL}) = 0
sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {SIG_DFL}) = 0
gettimeofday({948904719, 138951}, NULL) = 0
sendto(3, "\10\0D\201a\1\0\0\17#\2178\307\36"..., 64, 0, {sin_family=AF_INET,
sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 64
sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
sigaction(SIGALRM, {0x8049ba0, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
alarm(10) = 0
recvfrom(3, "E\0\0T\0005\0\0@\1|r\177\0\0\1\177"..., 192, 0,
{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84
gettimeofday({948904719, 160224}, NULL) = 0
recvfrom(3, "E\0\0T\0006\0\0\377\1\275p\177\0"..., 192, 0,
{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84
gettimeofday({948904719, 166952}, NULL) = 0
write(1, "64 bytes from 127.0.0.1: icmp_se"...,
5764 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=28.0 ms

Example 2
---------
strace passwd 2>&1 | grep open
produces the following output
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/opt/kde/lib/libc.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/libc.so.5", O_RDONLY) = 3
open("/dev", O_RDONLY) = 3
open("/var/run/utmp", O_RDONLY) = 3
open("/etc/passwd", O_RDONLY) = 3
open("/etc/shadow", O_RDONLY) = 3
open("/etc/login.defs", O_RDONLY) = 4
open("/dev/tty", O_RDONLY) = 4

The 2>&1 is done to redirect stderr to stdout & grep is then filtering this input
through the pipe for each line containing the string open.


Example 3
---------
Getting sophisticated
telnetd crashes & I don't know why

Steps
-----
1) Replace the following line in /etc/inetd.conf
telnet stream tcp nowait root /usr/sbin/in.telnetd -h
with
telnet stream tcp nowait root /blah

2) Create the file /blah with the following contents to start tracing telnetd
#!/bin/bash
/usr/bin/strace -o/t1 -f /usr/sbin/in.telnetd -h
3) chmod 700 /blah to make it executable only to root
4)
killall -HUP inetd
or ps aux | grep inetd
get inetd's process id
& kill -HUP inetd to restart it.

Important options
-----------------
-o is used to tell strace to output to a file in our case t1 in the root directory
-f is to follow children i.e.
e.g in our case above telnetd will start the login process & subsequently a shell like bash.
You will be able to tell which is which from the process ID's listed on the left hand side
of the strace output.
-p<pid> will tell strace to attach to a running process, yup this can be done provided
it isn't being traced or debugged already & you have enough privileges,
the reason 2 processes cannot trace or debug the same program is that strace
becomes the parent process of the one being debugged & processes ( unlike people )
can have only one parent.


However the file /t1 will get big quite quickly
to test it telnet 127.0.0.1

now look at what files in.telnetd execve'd
413 execve("/usr/sbin/in.telnetd", ["/usr/sbin/in.telnetd", "-h"], [/* 17 vars */]) = 0
414 execve("/bin/login", ["/bin/login", "-h", "localhost", "-p"], [/* 2 vars */]) = 0

Whey it worked!.


Other hints:
------------
If the program is not very interactive ( i.e. not much keyboard input )
& is crashing in one architecture but not in another you can do
an strace of both programs under as identical a scenario as you can
on both architectures outputting to a file then.
do a diff of the two traces using the diff program
i.e.
diff output1 output2
& maybe you'll be able to see where the call paths differed, this
is possibly near the cause of the crash.

More info
---------
Look at man pages for strace & the various syscalls
e.g. man strace, man alarm, man socket.


Performance Debugging
=====================
gcc is capable of compiling in profiling code just add the -p option
to the CFLAGS, this obviously affects program size & performance.
This can be used by the gprof gnu profiling tool or the
gcov the gnu code coverage tool ( code coverage is a means of testing
code quality by checking if all the code in an executable in exercised by
a tester ).


Using top to find out where processes are sleeping in the kernel
----------------------------------------------------------------
To do this copy the System.map from the root directory where
the linux kernel was built to the /boot directory on your
linux machine.
Start top
Now type fU<return>
You should see a new field called WCHAN which
tells you where each process is sleeping here is a typical output.

6:59pm up 41 min, 1 user, load average: 0.00, 0.00, 0.00
28 processes: 27 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: 0.0% user, 0.1% system, 0.0% nice, 99.8% idle
Mem: 254900K av, 45976K used, 208924K free, 0K shrd, 28636K buff
Swap: 0K av, 0K used, 0K free 8620K cached

PID USER PRI NI SIZE RSS SHARE WCHAN STAT LIB %CPU %MEM TIME COMMAND
750 root 12 0 848 848 700 do_select S 0 0.1 0.3 0:00 in.telnetd
767 root 16 0 1140 1140 964 R 0 0.1 0.4 0:00 top
1 root 8 0 212 212 180 do_select S 0 0.0 0.0 0:00 init
2 root 9 0 0 0 0 down_inte SW 0 0.0 0.0 0:00 kmcheck

The time command
----------------
Another related command is the time command which gives you an indication
of where a process is spending the majority of its time.
e.g.
time ping -c 5 nc
outputs
real 0m4.054s
user 0m0.010s
sys 0m0.010s

Debugging under VM
==================
Expand Down

0 comments on commit a6b42af

Please sign in to comment.