-
Notifications
You must be signed in to change notification settings - Fork 11
/
Copy pathHOWTO
174 lines (143 loc) · 7.31 KB
/
HOWTO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
The easiest way for using mtrace-ng is simply run a program under the control of mtrace-ng:
mtrace-ng -t -d 10 a.out
This will run a.out and records all memory allocations. When the program exits or mtrace-ng is terminated, it dumps all open memory allocations for each different stack backtrace:
.
.
.
Stack (mmap):
bytes used: 8392704
number of open allocations: 1
total number of allocations: 1
tsc: 4
[0x7f9e207abfd7] /lib64/libpthread.so.0 pthread_create
[0x4010e0] /home/stefani/mt/test/x.c:128 test
[0x40126a] /home/stefani/mt/test/x.c:189 main
[0x7f9e20427b55] /lib64/libc.so.6 __libc_start_main
[0x400b89] /home/stefani/mt/test/a.out _start
Stack (malloc):
bytes used: 4800
number of open allocations: 8
total number of allocations: 30
tsc: 75
[0x400dee] /home/stefani/mt/test/x.c:56 ThreadFuncA
[0x7f9e207ab3a4] /lib64/libpthread.so.0 __pthread_get_minstack
[0x7f9e204ee18d] /lib64/libc.so.6 clone
In this case you see for the last stack backtrace which calls malloc() by function ThreadFuncA in file /home/stefani/mt/test/x.c at line 56, which has 8 open allocations using a total of 4800 bytes.
The default sort order is "allocations", which sorts the output by the number of open allocations. The lowest value is printed out first and the highest value at last. So the most interesting values are always in view. If you wish a different sort order und can pass a -S option, for example.
mtrace-ng -t -d 10 -s usage a.out
This will sort by the total sum of bytes used:
Stack (malloc):
bytes used: 288
number of open allocations: 1
total number of allocations: 1
tsc: 1
[0x7ff0ca5562e2] /lib64/ld-linux-x86-64.so.2 _dl_mcount
[0x7ff0ca556a8e] /lib64/ld-linux-x86-64.so.2 _dl_allocate_tls
[0x7ff0c990c069] /lib64/libpthread.so.0 pthread_create
[0x401022] /home/stefani/mt/test/x.c:107 test
[0x40126a] /home/stefani/mt/test/x.c:189 main
[0x7ff0c9587b55] /lib64/libc.so.6 __libc_start_main
[0x400b89] /home/stefani/mt/test/a.out _start
.
.
.
Stack (mmap):
bytes used: 8392704
number of open allocations: 1
total number of allocations: 1
tsc: 2
[0x7ff0c990bfd7] /lib64/libpthread.so.0 pthread_create
[0x401081] /home/stefani/mt/test/x.c:117 test
[0x40126a] /home/stefani/mt/test/x.c:189 main
[0x7ff0c9587b55] /lib64/libc.so.6 __libc_start_main
[0x400b89] /home/stefani/mt/test/a.out _start
Stack (mmap):
bytes used: 8392704
number of open allocations: 1
total number of allocations: 1
tsc: 4
[0x7ff0c990bfd7] /lib64/libpthread.so.0 pthread_create
[0x4010e0] /home/stefani/mt/test/x.c:128 test
[0x40126a] /home/stefani/mt/test/x.c:189 main
[0x7ff0c9587b55] /lib64/libc.so.6 __libc_start_main
[0x400b89] /home/stefani/mt/test/a.out _start
So you can see that the last two stack backtraces both called mmap() from inside the pthread_create() function. But since there are two different backtrace stacks there will be not merged together (the first on is called by x.c:117 and the other by x.c:128). Lets assume for the above example the stack backtrace depth is set (to the not really useful) value of 1, then mtrace-ng would output:
Stack (mmap):
bytes used: 16785408
number of open allocations: 2
total number of allocations: 2
tsc: 4
[0x7ff0c990bfd7] /lib64/libpthread.so.0 pthread_create
In this case it is not possible to identifiy which function has been called pthread_create() and some important informations get lost. So it always reasonable to use a stack depth which allows to distinguish the callers. For C programs a depth value of 10 is recommended and for C++ a minimum of 20 is useful.
It is also possible to write the output to a file instead to stderr, for example:
mtrace-ng -t -d 10 -o /tmp/mtrace-ng.out a.out
In this case the dump will written to /tmp/mtrace-ng.out after the program exits or mtrace-ng is terminated. Note that in the case the sort order is inverted, the highest value is written first to the file and the lowest value at last. So the most interesting values are always in view when opening the file with an editor.
It is also possible to attach to an already running program:
mtrace-ng -t -d 10 -o /tmp/mtrace-ng.out -p $(pidof -s a.out)
This is e.g. useful when you not want do trace the whole startup of the program or you want to test only a specific function of a program. Imaging a test scenario where a leak is assumed. So attached mtrace-ng, run the tests (more then once is better) and then terminated mtrace-ng by Ctrl-C. One of last dump stack backtraces should show the culprit.
A more sophisticate way to find a memory leak is to scan for lost pointer values, e.g.
mtrace-ng -a -s leaks -d 10 -o /tmp/mtrace-ng.out -p $(pidof -s a.out)
After attach and run the tests and terminating mtrace-ng by Ctrl-C, the memory will be scanned for lost pointer. The dump output will then sorted by the number of leaked allocations:
process 12166 scanning 18 allocations
process 12166
leaks reported: 15
new leaks found: 15
leaked bytes: 8064
leaked at 0x02030010 (288 bytes)
leaked at 0x02030140 (288 bytes)
leaked at 0x02030270 (288 bytes)
leaked at 0x7fb4740008c0 (0 bytes)
leaked at 0x7fb4740008e0 (400 bytes)
leaked at 0x7fb474000a80 (800 bytes)
leaked at 0x7fb474000db0 (1200 bytes)
leaked at 0x7fb474001270 (0 bytes)
leaked at 0x7fb474001290 (400 bytes)
leaked at 0x7fb474001430 (800 bytes)
leaked at 0x7fb474001760 (1200 bytes)
leaked at 0x7fb474001c20 (0 bytes)
leaked at 0x7fb474001c40 (400 bytes)
leaked at 0x7fb474001de0 (800 bytes)
leaked at 0x7fb474002110 (1200 bytes)
leaks total: 15
.
.
.
Stack (malloc):
bytes used: 288
number of open allocations: 1
total number of allocations: 1
leaked allocations: 1 (288 bytes)
tsc: 6
[0x7fb47aae72e2] /lib64/ld-linux-x86-64.so.2 _dl_mcount
[0x7fb47aae7a8e] /lib64/ld-linux-x86-64.so.2 _dl_allocate_tls
[0x7fb479e9d069] /lib64/libpthread.so.0 pthread_create
[0x4010e0] /home/stefani/mt/test/x.c:128 test
[0x40126a] /home/stefani/mt/test/x.c:189 main
[0x7fb479b18b55] /lib64/libc.so.6 __libc_start_main
[0x400b89] /home/stefani/mt/test/a.out _start
Stack (malloc):
bytes used: 7200
number of open allocations: 12
total number of allocations: 45
leaked allocations: 12 (7200 bytes)
tsc: 110
[0x400dee] /home/stefani/mt/test/x.c:56 ThreadFuncA
[0x7fb479e9c3a4] /lib64/libpthread.so.0 __pthread_get_minstack
[0x7fb479bdf18d] /lib64/libc.so.6 clone
So you can see there are 15 leaked pointer in this case and the main culprit is a malloc() called from ThreadFuncA().
When no C++ Code is traced or when using the libstdc++ (which calls malloc() and free() inside the allocation operators), the breakpoints for the C++ allocators can be omitted by the --nocpp option. This increases the trace performance.
There is also a server mode and a client with an command line interface. This is useful when you want get information during the runtime
of a program, for example:
mtrace-ng -f -r 0.0.0.0 -w -s -d 10 a.out
This start the client waiting for a client connection before running the program a.out.
Then you can connect the client using an other terminal or on an other computer:
mtrace-ng -r 127.0.0.1
memtrace info:
follow fork: no
follow exec: no
verbose: no
do trace: yes
stack depth: 10
mtrace>
No you can ask for help with the command "help", or dump the current state of the bookkeeping by using the command "dump".
For more information about the interactive mode see the manual page mtrace-ng(1) section INTERACTIVE MODE.