-
Notifications
You must be signed in to change notification settings - Fork 0
/
RunningSanitizers.html
307 lines (261 loc) · 17.2 KB
/
RunningSanitizers.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
<!DOCTYPE html>
<html lang="en" data-content_root="./">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Running Sanitizers</title>
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
<link rel="stylesheet" type="text/css" href="_static/bootstrap-sphinx.css?v=fadd4351" />
<link rel="stylesheet" type="text/css" href="_static/custom.css?v=77160d70" />
<script src="_static/documentation_options.js?v=a8da1a53"></script>
<script src="_static/doctools.js?v=9bcbadda"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="MVP Design" href="MVPDesign.html" />
<link rel="prev" title="Testing Utilities" href="TestingUtilities.html" />
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-59110517-1', 'auto');
ga('send', 'pageview');
</script>
</head><body>
<div id="navbar" class="navbar navbar-default ">
<div class="container">
<div class="navbar-header">
<!-- .btn-navbar is used as the toggle for collapsed navbar content -->
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".nav-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="http://www.mantidproject.org">
</a>
<span class="navbar-text navbar-version pull-left"><b>main</b></span>
</div>
<div class="collapse navbar-collapse nav-collapse">
<ul class="nav navbar-nav">
<li class="divider-vertical"></li>
<li><a href="index.html">Home</a></li>
<li><a href="https://download.mantidproject.org">Download</a></li>
<li><a href="https://docs.mantidproject.org">User Documentation</a></li>
<li><a href="http://www.mantidproject.org/contact">Contact Us</a></li>
</ul>
<form class="navbar-form navbar-right" action="search.html" method="get">
<div class="form-group">
<input type="text" name="q" class="form-control" placeholder="Search" />
</div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div>
<p>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="nav-item nav-item-0"><a href="index.html">Documentation</a> »</li>
<li class="nav-item nav-item-this"><a href="">Running Sanitizers</a></li>
</ul>
</div> </p>
</div>
<div class="container">
<div class="row">
<div class="body col-md-12 content" role="main">
<section id="running-sanitizers">
<span id="runningsanitizers"></span><h1>Running Sanitizers<a class="headerlink" href="#running-sanitizers" title="Link to this heading">¶</a></h1>
<nav class="contents local" id="contents">
<ul class="simple">
<li><p><a class="reference internal" href="#overview" id="id1">Overview</a></p></li>
<li><p><a class="reference internal" href="#pre-requisites" id="id2">Pre-requisites</a></p></li>
<li><p><a class="reference internal" href="#switching-to-sanitizer-build" id="id3">Switching to Sanitizer Build</a></p>
<ul>
<li><p><a class="reference internal" href="#command-line" id="id4">Command Line</a></p></li>
<li><p><a class="reference internal" href="#cmake-gui" id="id5">CMake GUI</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#running-tests" id="id6">Running Tests</a></p></li>
<li><p><a class="reference internal" href="#visual-studio-address-sanitizer" id="id7">Visual Studio Address Sanitizer</a></p></li>
<li><p><a class="reference internal" href="#advanced-details" id="id8">Advanced Details</a></p>
<ul>
<li><p><a class="reference internal" href="#instrumenting-python-advanced" id="id9">Instrumenting Python (Advanced)</a></p></li>
</ul>
</li>
<li><p><a class="reference internal" href="#common-problems" id="id10">Common Problems</a></p>
<ul>
<li><p><a class="reference internal" href="#library-leaks-appearing" id="id11">Library Leaks Appearing</a></p></li>
<li><p><a class="reference internal" href="#asan-was-not-the-first-library-loaded" id="id12">ASAN was not the first library loaded</a></p></li>
</ul>
</li>
</ul>
</nav>
<section id="overview">
<h2><a class="toc-backref" href="#id1" role="doc-backlink">Overview</a><a class="headerlink" href="#overview" title="Link to this heading">¶</a></h2>
<p>Sanitizers allow developers to find any issues with their code at runtime
related to memory, threading or undefined behaviour.</p>
</section>
<section id="pre-requisites">
<h2><a class="toc-backref" href="#id2" role="doc-backlink">Pre-requisites</a><a class="headerlink" href="#pre-requisites" title="Link to this heading">¶</a></h2>
<p>The following tooling is required for the GCC compiler:</p>
<ul class="simple">
<li><p>GCC-8 onwards</p></li>
<li><p>Clang 3.2 onwards</p></li>
</ul>
<p>Microsoft Visual Studio 2019 provides an (experimental) address sanitizer that applies to x64 targets in version 16.7 and later</p>
</section>
<section id="switching-to-sanitizer-build">
<h2><a class="toc-backref" href="#id3" role="doc-backlink">Switching to Sanitizer Build</a><a class="headerlink" href="#switching-to-sanitizer-build" title="Link to this heading">¶</a></h2>
<p>The following sanitizers modes are available:</p>
<ul class="simple">
<li><p>Address</p></li>
<li><p>Thread (GCC only)</p></li>
<li><p>Undefined (GCC only)</p></li>
</ul>
<p>It’s recommended to build in <em>RelWithDebInfo</em> mode, CMake will automatically
switch to -O1 and append extra flags for stack traces to work correctly whilst
preserving high performance</p>
<p>If required, a build can be completed in debug mode too, however the
performance penalty will be pretty severe.</p>
<section id="command-line">
<h3><a class="toc-backref" href="#id4" role="doc-backlink">Command Line</a><a class="headerlink" href="#command-line" title="Link to this heading">¶</a></h3>
<p>First, delete <em>CMakeCache.txt</em> if using GCC and your system compiler is GCC 7 or below
(you can check with <em>gcc -v</em>).</p>
<p>To change to a sanitized build navigate to your build folder and execute the
following if you’re using a single-config generator (eg Ninja):</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>cmake<span class="w"> </span>*path_to_src*<span class="w"> </span>-DUSE_SANITIZER<span class="o">=</span>*Mode*<span class="w"> </span>-DCMAKE_BUILD_TYPE<span class="o">=</span>RelWithDebInfo
</pre></div>
</div>
<p>…or the following with a multi-config generator (eg Visual Studio):</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>cmake<span class="w"> </span>*path_to_src*<span class="w"> </span>-DUSE_SANITIZER<span class="o">=</span>*Mode*
</pre></div>
</div>
<p>If you are using GCC and need to specify a different compiler too (for example if your system
default is GCC-7)</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nv">CC</span><span class="o">=</span>gcc-8<span class="w"> </span><span class="nv">CXX</span><span class="o">=</span>g++-8<span class="w"> </span>cmake<span class="w"> </span>*path_to_src*<span class="w"> </span>-DUSE_SANITIZER<span class="o">=</span>*Mode*<span class="w"> </span>-DCMAKE_BUILD_TYPE<span class="o">=</span>RelWithDebInfo
</pre></div>
</div>
<p>For example, to switch to an address sanitizer build the following can be used:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>cmake<span class="w"> </span>*path_to_src*<span class="w"> </span>-DUSE_SANITIZER<span class="o">=</span>Address<span class="w"> </span>-DCMAKE_BUILD_TYPE<span class="o">=</span>RelWithDebInfo
</pre></div>
</div>
</section>
<section id="cmake-gui">
<h3><a class="toc-backref" href="#id5" role="doc-backlink">CMake GUI</a><a class="headerlink" href="#cmake-gui" title="Link to this heading">¶</a></h3>
<ul class="simple">
<li><p>If using GCC, delete the cache if your system compiler is GCC 7 or below (you can check
with <em>gcc -v</em>)</p></li>
<li><p>Hit configure</p></li>
<li><p>If using GCC, click specify native compilers to <em>gcc-8</em> and <em>g++-8</em></p></li>
<li><p>If using a single-config generator, change <em>CMAKE_BUILD_TYPE</em> to <em>RelWithDebInfo</em> (or <em>Debug</em>)</p></li>
<li><p>Change <em>USE_SANITIZERS</em> to any of the options in the drop down</p></li>
<li><p>Hit configure again, then generate and rebuild the project</p></li>
</ul>
</section>
</section>
<section id="running-tests">
<h2><a class="toc-backref" href="#id6" role="doc-backlink">Running Tests</a><a class="headerlink" href="#running-tests" title="Link to this heading">¶</a></h2>
<p>Several upstream linked libraries currently contain leaks which we cannot
resolve (or have been resolved but not appeared downstream).</p>
<p>We can suppress warnings in the address sanitizer by setting environment
variables in the console before running in each mode.</p>
</section>
<section id="visual-studio-address-sanitizer">
<h2><a class="toc-backref" href="#id7" role="doc-backlink">Visual Studio Address Sanitizer</a><a class="headerlink" href="#visual-studio-address-sanitizer" title="Link to this heading">¶</a></h2>
<p>The Visual Studio address sanitizer raises exceptions with code 0xC0000005 as part of its normal operation.
The exceptions are handled but they cause the debugger to stop. These exceptions need to be ignored in the Debug, Windows, Exception Settings dialog</p>
<p>Genuine sanitizer issues cause an unhandled exception and the debugger will stop at the relevant unitline.</p>
<p>Some parts of the Mantid code (eg H5Util.cpp) don’t compile when the address sanitizer is enabled and the /O2 optimisation flag is used (in RelWithDebInfo).
This flag is switched to /O1 in order to improve stack trace information (see above) and this fortunately removes the compilation errors.</p>
<p>The following path (or equivalent for your Visual Studio version) needs to be added to your path environment variable in order for Visual Studio to find some .lib files that are used by the sanitizer and also to locate a symbolizer exe that is required for useful error messages:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>C:<span class="se">\\</span>Program<span class="w"> </span>Files<span class="w"> </span><span class="o">(</span>x86<span class="o">)</span><span class="se">\\</span>Microsoft<span class="w"> </span>Visual<span class="w"> </span>Studio<span class="se">\\</span><span class="m">2019</span><span class="se">\\</span>Community<span class="se">\\</span>VC<span class="se">\\</span>Tools<span class="se">\\</span>MSVC<span class="se">\\</span><span class="m">14</span>.27.29110<span class="se">\\</span>bin<span class="se">\\</span>Hostx64<span class="se">\\</span>x64
</pre></div>
</div>
<p>Additional information is available on this <a class="reference external" href="https://devblogs.microsoft.com/cppblog/asan-for-windows-x64-and-debug-build-support/">web page</a>.</p>
</section>
<section id="advanced-details">
<h2><a class="toc-backref" href="#id8" role="doc-backlink">Advanced Details</a><a class="headerlink" href="#advanced-details" title="Link to this heading">¶</a></h2>
<p>Most developers do not need to read this, but it’s good for those who
want to know what’s happening</p>
<p>CMake substitutes in various flags for the address sanitizer builds to
setup suppressions etc… this is the equivalent of doing the following
in a local shell:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="nb">export</span><span class="w"> </span><span class="nv">ASAN_OPTIONS</span><span class="o">=</span><span class="s2">"verify_asan_link_order=0:detect_stack_use_after_return=true:halt_on_error=false:suppressions=*path_to_mantid*/tools/Sanitizer/Address.supp"</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">LSAN_OPTIONS</span><span class="o">=</span><span class="s2">"suppressions=*path_to_mantid*/tools/Sanitizer/Leak.supp"</span>
</pre></div>
</div>
<p>All code executed which is executed in that shell will now be sanitized
correctly. To save developers effort the CXX_ADD_TEST macro (in
FindCxxTest.cmake) will append these environment variables on a developers
behalf.</p>
<section id="instrumenting-python-advanced">
<h3><a class="toc-backref" href="#id9" role="doc-backlink">Instrumenting Python (Advanced)</a><a class="headerlink" href="#instrumenting-python-advanced" title="Link to this heading">¶</a></h3>
<p>Currently any code started in Python (i.e. Python Unit Tests) will not pre-load
ASAN instrumentation. This can be split into two categories:</p>
<ul class="simple">
<li><p>Code which uses Python only components: Not worth instrumenting as any
issues will be upstream. This also will emit an error if
<em>verify_asan_link_order</em> is set to true, as we technically haven’t
instrumented anything (unless you have a sanitized Python build)</p></li>
<li><p>Code which uses Mantid C++ components: This can be instrumented, but
(currently) isn’t by default, as the user has to determine the <em>LD_PRELOAD</em>
path.</p></li>
</ul>
<p>If you need / want to profile C++ components which are triggered from Python
the following steps should setup your environment:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># Get the path to your linked ASAN</span>
ldd<span class="w"> </span>bin/KernelTest<span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"libasan"</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">LD_PRELOAD</span><span class="o">=</span>/usr/lib/path_to/libasan.so.x
<span class="c1"># leak detection should only show the largest 25 leaks</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">LSAN_OPTIONS</span><span class="o">=</span><span class="s2">"max_leaks=25"</span>
<span class="c1"># You may want to re-run the ASAN_OPTIONS export dropping</span>
<span class="c1"># the verify to make sure that the C++ component is being instrumented:</span>
<span class="c1"># log_path is the prefix for the file the results are written to</span>
<span class="nb">export</span><span class="w"> </span><span class="nv">ASAN_OPTIONS</span><span class="o">=</span><span class="s2">"detect_stack_use_after_return=true:halt_on_error=false:log_path=asan:suppressions=*path_to_mantid*/tools/Sanitizer/Address.supp"</span>
</pre></div>
</div>
</section>
</section>
<section id="common-problems">
<h2><a class="toc-backref" href="#id10" role="doc-backlink">Common Problems</a><a class="headerlink" href="#common-problems" title="Link to this heading">¶</a></h2>
<section id="library-leaks-appearing">
<h3><a class="toc-backref" href="#id11" role="doc-backlink">Library Leaks Appearing</a><a class="headerlink" href="#library-leaks-appearing" title="Link to this heading">¶</a></h3>
<p>Check that you have correctly spelt <em>suppressions</em> as there will be no warnings
for typos. A good check is to put some random characters in the .supp files,
which will cause all tests to fail if it’s begin read.</p>
<p>Any new third party memory leaks need to go into <em>Leaks.supp</em> not
<em>Address.supp</em> (which should ideally be completely empty) to be suppressed.</p>
</section>
<section id="asan-was-not-the-first-library-loaded">
<h3><a class="toc-backref" href="#id12" role="doc-backlink">ASAN was not the first library loaded</a><a class="headerlink" href="#asan-was-not-the-first-library-loaded" title="Link to this heading">¶</a></h3>
<p>This can appear when running Python tests, as the executable is not build
with instrumentation. To avoid this warning ensure that
<em>verify_asan_link_order=0</em> is set in your environment and that you are
using GCC 8 onwards.</p>
</section>
</section>
</section>
</div>
</div>
</div>
<footer class="footer">
<div class="container">
<ul class="nav navbar-nav" style=" float: right;">
<li>
<a href="TestingUtilities.html" title="Previous Chapter: Testing Utilities"><span class="glyphicon glyphicon-chevron-left visible-sm"></span><span class="hidden-sm hidden-tablet">« Testing Utilities</span>
</a>
</li>
<li>
<a href="MVPDesign.html" title="Next Chapter: MVP Design"><span class="glyphicon glyphicon-chevron-right visible-sm"></span><span class="hidden-sm hidden-tablet">MVP Design »</span>
</a>
</li>
<li><a href="#">Back to top</a></li>
</ul>
<p>
</p>
</div>
</footer>
</body>
</html>