-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Documenter.jl
committed
Dec 15, 2023
1 parent
cc9a0a5
commit 8dd8372
Showing
10 changed files
with
11 additions
and
11 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,3 @@ | ||
| <!DOCTYPE html> | ||
| <html lang="en"><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width, initial-scale=1.0"/><title>Running your Experiments · Experimenter.jl</title><script data-outdated-warner src="../assets/warner.js"></script><link rel="canonical" href="https://JamieMair.github.io/Experimenter.jl/execution/"/><link href="https://cdnjs.cloudflare.com/ajax/libs/lato-font/3.0.0/css/lato-font.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/juliamono/0.045/juliamono.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/fontawesome.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/solid.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/brands.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.13.24/katex.min.css" rel="stylesheet" type="text/css"/><script>documenterBaseURL=".."</script><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js" data-main="../assets/documenter.js"></script><script src="../siteinfo.js"></script><script src="../../versions.js"></script><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../assets/themes/documenter-dark.css" data-theme-name="documenter-dark" data-theme-primary-dark/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../assets/themes/documenter-light.css" data-theme-name="documenter-light" data-theme-primary/><script src="../assets/themeswap.js"></script></head><body><div id="documenter"><nav class="docs-sidebar"><div class="docs-package-name"><span class="docs-autofit"><a href="../">Experimenter.jl</a></span></div><form class="docs-search" action="../search/"><input class="docs-search-query" id="documenter-search-query" name="q" type="text" placeholder="Search docs"/></form><ul class="docs-menu"><li><a class="tocitem" href="../">Home</a></li><li><a class="tocitem" href="../getting_started/">Getting Started</a></li><li class="is-active"><a class="tocitem" href>Running your Experiments</a><ul class="internal"><li><a class="tocitem" href="#Executing-in-Parallel"><span>Executing in Parallel</span></a></li></ul></li><li><a class="tocitem" href="../distributed/">Distributed Execution</a></li><li><a class="tocitem" href="../store/">Data Store</a></li><li><a class="tocitem" href="../snapshots/">Custom Snapshots</a></li><li><a class="tocitem" href="../clusters/">Cluster Support</a></li><li><a class="tocitem" href="../api/">Public API</a></li></ul><div class="docs-version-selector field has-addons"><div class="control"><span class="docs-label button is-static is-size-7">Version</span></div><div class="docs-selector control is-expanded"><div class="select is-fullwidth is-size-7"><select id="documenter-version-selector"></select></div></div></div></nav><div class="docs-main"><header class="docs-navbar"><nav class="breadcrumb"><ul class="is-hidden-mobile"><li class="is-active"><a href>Running your Experiments</a></li></ul><ul class="is-hidden-tablet"><li class="is-active"><a href>Running your Experiments</a></li></ul></nav><div class="docs-right"><a class="docs-edit-link" href="https://github.com/JamieMair/Experimenter.jl/blob/main/docs/src/execution.md" title="Edit on GitHub"><span class="docs-icon fab"></span><span class="docs-label is-hidden-touch">Edit on GitHub</span></a><a class="docs-settings-button fas fa-cog" id="documenter-settings-button" href="#" title="Settings"></a><a class="docs-sidebar-button fa fa-bars is-hidden-desktop" id="documenter-sidebar-button" href="#"></a></div></header><article class="content" id="documenter-page"><h1 id="Running-your-Experiments"><a class="docs-heading-anchor" href="#Running-your-Experiments">Running your Experiments</a><a id="Running-your-Experiments-1"></a><a class="docs-heading-anchor-permalink" href="#Running-your-Experiments" title="Permalink"></a></h1><p>Once you have created an experiment you can run it with the <code>@execute</code> macro supplied by <code>Experimenter.jl</code>, suppose you already have an experiment stored in the <code>experiment</code> variable and a database opened with the variable <code>db</code>, then you can execute simply with:</p><pre><code class="language-julia hljs">@execute experiment db SerialMode</code></pre><p>Which will only execute trials from the experiment that have not been completed. It is up to you to implement how to continue your simulations from snapshots, using the Snapshots API. </p><h2 id="Executing-in-Parallel"><a class="docs-heading-anchor" href="#Executing-in-Parallel">Executing in Parallel</a><a id="Executing-in-Parallel-1"></a><a class="docs-heading-anchor-permalink" href="#Executing-in-Parallel" title="Permalink"></a></h2><p>There are two main ways of executing your experiments in parallel: multithreading (Threads) or multiprocessing (Distributed). The former has lower latency, but the latter scales to working on across a cluster. The easiest option if you are executing on a single computer, use:</p><pre><code class="language-julia hljs">@execute experiment db MultithreadedMode</code></pre><p>By default, this will use as many threads as you have enabled. You can set this using the environment variable <code>JULIA_NUM_THREADS</code>, or by starting Julia with <code>--threads=X</code>, replacing <code>X</code> with the number you want. You can check what your current setting is with <code>Threads.nthreads()</code>.</p><p>On a cluster, we can change the execution mode to <code>DistributedMode</code>:</p><pre><code class="language-julia hljs">@execute experiment db DistributedMode</code></pre><p>This internally uses <code>pmap</code> from the <code>Distributed.jl</code> standard library, parallelising across all open workers. You can check the number of distributed workers with:</p><pre><code class="language-julia hljs">using Distributed | ||
| nworkers()</code></pre><p><code>Experimenter.jl</code> will not spin up processes for you, this is something you have to do yourself, see <a href="../distributed/#Distributed-Execution">Distributed Execution</a> for an in depth example. </p><h3 id="Heterogeneous-Execution"><a class="docs-heading-anchor" href="#Heterogeneous-Execution">Heterogeneous Execution</a><a id="Heterogeneous-Execution-1"></a><a class="docs-heading-anchor-permalink" href="#Heterogeneous-Execution" title="Permalink"></a></h3><p>If you want each distributed worker to be able to run multiple jobs at the same time, you can select a heterogeneous execution scheduling mode, which will allow each worker to run multiple trials simulatenously using multithreading. An example use case for this is where you have multiple nodes, each with many cores, and you do not wish to pay the memory cost from each separate process. Additionally, you can load data in a single process which can be reused by each execution in the same process. This mode may also allow multiple trials to share resources, such as a GPU, which typically only supports one process.</p><p>To run this, you simple change the mode to the <code>HeterogeneousMode</code> option, providing the number of threads to use on each worker, e.g.</p><pre><code class="language-julia hljs">@execute experiment db HeterogeneousMode(2)</code></pre><p>which will allow each distributed worker to run two trials simulatenously via multithreading. If this option is selected, it is encouraged that you enable multiple threads per worker when launching the process, e.g. with <code>addprocs</code>:</p><pre><code class="language-julia hljs">addprocs(4; exeflags=["--threads=2"])</code></pre><p>Otherwise, each worker may only have access to a single thread and the overall performance throughput will be worse.</p><p><!– TODO: Update the manifest to point to current experimenter version in docs –></p></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../getting_started/">« Getting Started</a><a class="docs-footer-nextpage" href="../distributed/">Distributed Execution »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 0.27.25 on <span class="colophon-date" title="Friday 15 December 2023 15:18">Friday 15 December 2023</span>. Using Julia version 1.9.4.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html> | ||
| nworkers()</code></pre><p><code>Experimenter.jl</code> will not spin up processes for you, this is something you have to do yourself, see <a href="../distributed/#Distributed-Execution">Distributed Execution</a> for an in depth example. </p><h3 id="Heterogeneous-Execution"><a class="docs-heading-anchor" href="#Heterogeneous-Execution">Heterogeneous Execution</a><a id="Heterogeneous-Execution-1"></a><a class="docs-heading-anchor-permalink" href="#Heterogeneous-Execution" title="Permalink"></a></h3><p>If you want each distributed worker to be able to run multiple jobs at the same time, you can select a heterogeneous execution scheduling mode, which will allow each worker to run multiple trials simulatenously using multithreading. An example use case for this is where you have multiple nodes, each with many cores, and you do not wish to pay the memory cost from each separate process. Additionally, you can load data in a single process which can be reused by each execution in the same process. This mode may also allow multiple trials to share resources, such as a GPU, which typically only supports one process.</p><p>To run this, you simple change the mode to the <code>HeterogeneousMode</code> option, providing the number of threads to use on each worker, e.g.</p><pre><code class="language-julia hljs">@execute experiment db HeterogeneousMode(2)</code></pre><p>which will allow each distributed worker to run two trials simulatenously via multithreading. If this option is selected, it is encouraged that you enable multiple threads per worker when launching the process, e.g. with <code>addprocs</code>:</p><pre><code class="language-julia hljs">addprocs(4; exeflags=["--threads=2"])</code></pre><p>Otherwise, each worker may only have access to a single thread and the overall performance throughput will be worse.</p><p><!– TODO: Update the manifest to point to current experimenter version in docs –></p></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../getting_started/">« Getting Started</a><a class="docs-footer-nextpage" href="../distributed/">Distributed Execution »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 0.27.25 on <span class="colophon-date" title="Friday 15 December 2023 15:27">Friday 15 December 2023</span>. Using Julia version 1.9.4.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html> |
Oops, something went wrong.