Upgrade MPIs on linux

This commit is contained in:
Bassem Girgis
2025-03-19 12:53:34 -05:00
parent d29ef955f5
commit a6198ecc7b
5503 changed files with 962014 additions and 56494 deletions

View File

@@ -0,0 +1,283 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8.9. Deprecated options &mdash; PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="9. Notifications" href="../notifications.html" />
<link rel="prev" title="8.8. Rankfiles" href="rankfiles.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">8. Process placement</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="overview.html">8.1. Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-slot">8.2. Definition of ‘slot’</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l2"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l2"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l2"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">8. </span>Process placement</a></li>
<li class="breadcrumb-item active"><span class="section-number">8.9. </span>Deprecated options</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/deprecated.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="deprecated-options">
<h1><span class="section-number">8.9. </span>Deprecated options<a class="headerlink" href="#deprecated-options" title="Permalink to this heading"></a></h1>
<p>These deprecated options will be removed in a future release.</p>
<table class="colwidths-given docutils align-default">
<colgroup>
<col style="width: 29%" />
<col style="width: 29%" />
<col style="width: 43%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Deprecated Option</p></th>
<th class="head"><p>Replacement</p></th>
<th class="head"><p>Description</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--bind-to-core</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">core</span></code></p></td>
<td><p>Bind processes to cores</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--bind-to-socket</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">package</span></code></p></td>
<td><p>Bind processes to processor sockets</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--bycore</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">core</span></code></p></td>
<td><p>Map processes by core</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--bynode</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">node</span></code></p></td>
<td><p>Launch processes one per node, cycling by node in a round-robin
fashion. This spreads processes evenly among nodes and assigns
ranks in a round-robin, “by node” manner.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--byslot</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">slot</span></code></p></td>
<td><p>Map and rank processes round-robin by slot</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--cpus-per-proc</span> <span class="pre">&lt;#perproc&gt;</span></code></p></td>
<td><p><cite>–map-by &lt;obj&gt;:PE=&lt;#perproc&gt;`</cite></p></td>
<td><p>Bind each process to the specified number of CPUs</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--cpus-per-rank</span> <span class="pre">&lt;#perrank&gt;</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">&lt;obj&gt;:PE=&lt;#perrank&gt;</span></code></p></td>
<td><p>Alias for <code class="docutils literal notranslate"><span class="pre">--cpus-per-proc</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--display-allocation</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--display</span> <span class="pre">ALLOC</span></code></p></td>
<td><p>Display the detected resource allocation</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">-display-devel-map</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--display</span> <span class="pre">MAP-DEVEL</span></code></p></td>
<td><p>Display a detailed process map (mostly intended for developers)
just before launch.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--display-map</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--display</span> <span class="pre">MAP</span></code></p></td>
<td><p>Display a table showing the mapped location of each process
prior to launch.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--display-topo</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--display</span> <span class="pre">TOPO</span></code></p></td>
<td><p>Display the topology as part of the process map (mostly
intended for developers) just before launch.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--do-not-launch</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">:DONOTLAUNCH</span></code></p></td>
<td><p>Perform all necessary operations to prepare to launch the
application, but do not actually launch it (usually used to
test mapping patterns).</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--do-not-resolve</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">:DONOTRESOLVE</span></code></p></td>
<td><p>Do not attempt to resolve interfaces — usually used to
determine proposed process placement/binding prior to obtaining
an allocation.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">-N</span> <span class="pre">&lt;num&gt;</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">prr:&lt;num&gt;:node</span></code></p></td>
<td><p>Launch <code class="docutils literal notranslate"><span class="pre">num</span></code> processes per node on all allocated nodes</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--nolocal</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">:NOLOCAL</span></code></p></td>
<td><p>Do not run any copies of the launched application on the same
node as <code class="docutils literal notranslate"><span class="pre">prun</span></code> is running. This option will override listing
the <code class="docutils literal notranslate"><span class="pre">localhost</span></code> with <code class="docutils literal notranslate"><span class="pre">--host</span></code> or any other host-specifying
mechanism.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--nooversubscribe</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">:NOOVERSUBSCRIBE</span></code></p></td>
<td><p>Do not oversubscribe any nodes; error (without starting any
processes) if the requested number of processes would cause
oversubscription. This option implicitly sets “max_slots” equal
to the “slots” value for each node. (Enabled by default).</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--npernode</span> <span class="pre">&lt;#pernode&gt;</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">ppr:&lt;#pernode&gt;:node</span></code></p></td>
<td><p>On each node, launch this many processes</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--npersocket</span> <span class="pre">&lt;#persocket&gt;</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">ppr:&lt;#perpackage&gt;:package</span></code></p></td>
<td><p>On each node, launch this many processes times the number of
processor sockets on the node. The <code class="docutils literal notranslate"><span class="pre">--npersocket</span></code> option also
turns on the <code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">socket</span></code> option. The term <code class="docutils literal notranslate"><span class="pre">socket</span></code>
has been globally replaced with <code class="docutils literal notranslate"><span class="pre">package</span></code>.</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--oversubscribe</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">:OVERSUBSCRIBE</span></code></p></td>
<td><p>Nodes are allowed to be oversubscribed, even on a managed
system, and overloading of processing elements.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--pernode</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">ppr:1:node</span></code></p></td>
<td><p>On each node, launch one process</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--ppr</span></code></p></td>
<td><p><cite>–map-by ppr:&lt;list&gt;`</cite></p></td>
<td><p>Comma-separated list of number of processes on a given resource type
[default: <code class="docutils literal notranslate"><span class="pre">none</span></code>].</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--rankfile</span> <span class="pre">&lt;FILENAME&gt;</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">rankfile:FILE=&lt;FILENAME&gt;</span></code></p></td>
<td><p>Use a rankfile for mapping/ranking/binding</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--report-bindings</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--display</span> <span class="pre">BINDINGS</span></code></p></td>
<td><p>Report any bindings for launched processes</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--tag-output</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--output</span> <span class="pre">TAG</span></code></p></td>
<td><p>Tag all output with <code class="docutils literal notranslate"><span class="pre">[job,rank]</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--timestamp-output</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--output</span> <span class="pre">TIMESTAMP</span></code></p></td>
<td><p>Timestamp all application process output</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--use-hwthread-cpus</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">:HWTCPUS</span></code></p></td>
<td><p>Use hardware threads as independent CPUs</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--xml</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">--output</span> <span class="pre">XML</span></code></p></td>
<td><p>Provide all output in XML format</p></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="rankfiles.html" class="btn btn-neutral float-left" title="8.8. Rankfiles" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="../notifications.html" class="btn btn-neutral float-right" title="9. Notifications" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>

View File

@@ -0,0 +1,201 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8.7. Diagnostics &mdash; PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="8.8. Rankfiles" href="rankfiles.html" />
<link rel="prev" title="8.6. Overloading and Oversubscribing" href="limits.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">8. Process placement</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="overview.html">8.1. Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-slot">8.2. Definition of ‘slot’</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l2"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">8.7. Diagnostics</a></li>
<li class="toctree-l2"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">8. </span>Process placement</a></li>
<li class="breadcrumb-item active"><span class="section-number">8.7. </span>Diagnostics</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/diagnostics.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="diagnostics">
<h1><span class="section-number">8.7. </span>Diagnostics<a class="headerlink" href="#diagnostics" title="Permalink to this heading"></a></h1>
<p>PRRTE provides various diagnostic reports that aid the user in
verifying and tuning the mapping/ranking/binding for a specific job.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">:REPORT</span></code> qualifier to the <code class="docutils literal notranslate"><span class="pre">--bind-to</span></code> command line option can
be used to report process bindings.</p>
<p>As an example, consider a node with:</p>
<ul class="simple">
<li><p>2 processor packages,</p></li>
<li><p>4 cores per package, and</p></li>
<li><p>8 hardware threads per core.</p></li>
</ul>
<p>In each of the examples below the binding is reported in a human readable
format.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ prun --np 4 --map-by core --bind-to core:REPORT ./a.out
[node01:103137] MCW rank 0 bound to package[0][core:0]
[node01:103137] MCW rank 1 bound to package[0][core:1]
[node01:103137] MCW rank 2 bound to package[0][core:2]
[node01:103137] MCW rank 3 bound to package[0][core:3]
</pre></div>
</div>
<p>In the example above, processes are bound to successive cores on the
first package.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ prun --np 4 --map-by package --bind-to package:REPORT ./a.out
[node01:103115] MCW rank 0 bound to package[0][core:0-9]
[node01:103115] MCW rank 1 bound to package[1][core:10-19]
[node01:103115] MCW rank 2 bound to package[0][core:0-9]
[node01:103115] MCW rank 3 bound to package[1][core:10-19]
</pre></div>
</div>
<p>In the example above, processes are bound to all cores on successive
packages in a round-robin fashion.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ prun --np 4 --map-by package:PE=2 --bind-to core:REPORT ./a.out
[node01:103328] MCW rank 0 bound to package[0][core:0-1]
[node01:103328] MCW rank 1 bound to package[1][core:10-11]
[node01:103328] MCW rank 2 bound to package[0][core:2-3]
[node01:103328] MCW rank 3 bound to package[1][core:12-13]
</pre></div>
</div>
<p>The example above shows us that 2 cores have been bound per process.
The <code class="docutils literal notranslate"><span class="pre">:PE=2</span></code> qualifier states that 2 CPUs underneath the package
(which would be cores in this case) are mapped to each process.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ prun --np 4 --map-by core:PE=2:HWTCPUS --bind-to :REPORT hostname
[node01:103506] MCW rank 0 bound to package[0][hwt:0-1]
[node01:103506] MCW rank 1 bound to package[0][hwt:8-9]
[node01:103506] MCW rank 2 bound to package[0][hwt:16-17]
[node01:103506] MCW rank 3 bound to package[0][hwt:24-25]
</pre></div>
</div>
<p>The example above shows us that 2 hardware threads have been bound per
process. In this case <code class="docutils literal notranslate"><span class="pre">prun</span></code> is directing the DVM to map by
hardware threads since we used the <code class="docutils literal notranslate"><span class="pre">:HWTCPUS</span></code> qualifier. Without
that qualifier this command would return an error since by default the
DVM will not map to resources smaller than a core. The <code class="docutils literal notranslate"><span class="pre">:PE=2</span></code>
qualifier states that 2 processing elements underneath the core (which
would be hardware threads in this case) are mapped to each process.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ prun --np 4 --bind-to none:REPORT hostname
[node01:107126] MCW rank 0 is not bound (or bound to all available processors)
[node01:107126] MCW rank 1 is not bound (or bound to all available processors)
[node01:107126] MCW rank 2 is not bound (or bound to all available processors)
[node01:107126] MCW rank 3 is not bound (or bound to all available processors)
</pre></div>
</div>
<p>Binding is turned off in the above example, as reported.</p>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="limits.html" class="btn btn-neutral float-left" title="8.6. Overloading and Oversubscribing" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="rankfiles.html" class="btn btn-neutral float-right" title="8.8. Rankfiles" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>

View File

@@ -0,0 +1,506 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8.4. Examples &mdash; PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="8.5. Fundamentals" href="fundamentals.html" />
<link rel="prev" title="8.1. Overview" href="overview.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">8. Process placement</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="overview.html">8.1. Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-slot">8.2. Definition of ‘slot’</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">8.4. Examples</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#specifying-host-nodes">8.4.1. Specifying Host Nodes</a></li>
<li class="toctree-l3"><a class="reference internal" href="#process-mapping-ranking-binding-options">8.4.2. Process Mapping / Ranking / Binding Options</a></li>
<li class="toctree-l3"><a class="reference internal" href="#id1">8.4.3. Specifying Host Nodes</a></li>
<li class="toctree-l3"><a class="reference internal" href="#specifying-number-of-processes">8.4.4. Specifying Number of Processes</a></li>
<li class="toctree-l3"><a class="reference internal" href="#mapping-processes-to-nodes-using-policies">8.4.5. Mapping Processes to Nodes Using Policies</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l2"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l2"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l2"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">8. </span>Process placement</a></li>
<li class="breadcrumb-item active"><span class="section-number">8.4. </span>Examples</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/examples.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="examples">
<h1><span class="section-number">8.4. </span>Examples<a class="headerlink" href="#examples" title="Permalink to this heading"></a></h1>
<p>Listed here are the subset of command line options that will be used
in the process mapping/ranking/binding examples below.</p>
<div class="section" id="specifying-host-nodes">
<h2><span class="section-number">8.4.1. </span>Specifying Host Nodes<a class="headerlink" href="#specifying-host-nodes" title="Permalink to this heading"></a></h2>
<p>Use one of the following options to specify which hosts (nodes) within
the PRRTE DVM environment to run on.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">--</span><span class="n">host</span> <span class="o">&lt;</span><span class="n">host1</span><span class="p">,</span><span class="n">host2</span><span class="p">,</span><span class="o">...</span><span class="p">,</span><span class="n">hostN</span><span class="o">&gt;</span>
<span class="c1"># or</span>
<span class="o">--</span><span class="n">host</span> <span class="o">&lt;</span><span class="n">host1</span><span class="p">:</span><span class="n">X</span><span class="p">,</span><span class="n">host2</span><span class="p">:</span><span class="n">Y</span><span class="p">,</span><span class="o">...</span><span class="p">,</span><span class="n">hostN</span><span class="p">:</span><span class="n">Z</span><span class="o">&gt;</span>
</pre></div>
</div>
<ul class="simple">
<li><p>List of hosts on which to invoke processes. After each hostname a
colon (<code class="docutils literal notranslate"><span class="pre">:</span></code>) followed by a positive integer can be used to specify
the number of slots on that host (<code class="docutils literal notranslate"><span class="pre">:X</span></code>, <code class="docutils literal notranslate"><span class="pre">:Y</span></code>, and <code class="docutils literal notranslate"><span class="pre">:Z</span></code>). The
default is <code class="docutils literal notranslate"><span class="pre">1</span></code>.</p></li>
</ul>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">--</span><span class="n">hostfile</span> <span class="o">&lt;</span><span class="n">hostfile</span><span class="o">&gt;</span>
</pre></div>
</div>
<ul class="simple">
<li><p>Provide a hostfile to use.</p></li>
</ul>
</div>
<div class="section" id="process-mapping-ranking-binding-options">
<h2><span class="section-number">8.4.2. </span>Process Mapping / Ranking / Binding Options<a class="headerlink" href="#process-mapping-ranking-binding-options" title="Permalink to this heading"></a></h2>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">-c</span> <span class="pre">#</span></code>, <code class="docutils literal notranslate"><span class="pre">-n</span> <span class="pre">#</span></code>, <code class="docutils literal notranslate"><span class="pre">--n</span> <span class="pre">#</span></code>, <code class="docutils literal notranslate"><span class="pre">--np</span> <span class="pre">&lt;#&gt;</span></code>: Run this many copies of
the program on the given nodes. This option indicates that the
specified file is an executable program and not an application
context. If no value is provided for the number of copies to execute
(i.e., neither the <code class="docutils literal notranslate"><span class="pre">-np</span></code> nor its synonyms are provided on the
command line), <code class="docutils literal notranslate"><span class="pre">prun</span></code> will automatically execute a copy of the
program on each process slot (see below for description of a
“process slot”). This feature, however, can only be used in the SPMD
model and will return an error (without beginning execution of the
application) otherwise.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>These options specify the number of processes to launch.
None of the options imply a particular binding policy
— e.g., requesting <code class="docutils literal notranslate"><span class="pre">N</span></code> processes for each package
does not imply that the processes will be bound to the
package.</p>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">&lt;object&gt;</span></code>: Map to the specified object. Supported
objects include:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">slot</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">hwthread</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">core</span></code> (default)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">l1cache</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">l2cache</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">l3cache</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">numa</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">package</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">node</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">seq</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ppr</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">rankfile</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">pe-list</span></code></p></li>
</ul>
<p>Any object can include qualifiers by adding a colon (<code class="docutils literal notranslate"><span class="pre">:</span></code>) and any
colon-delimited combination of one or more of the following to the
<code class="docutils literal notranslate"><span class="pre">--map-by</span></code> options:</p>
<ul>
<li><p><code class="docutils literal notranslate"><span class="pre">PE=n</span></code> bind <code class="docutils literal notranslate"><span class="pre">n</span></code> processing elements to each process (can not
be used in combination with rankfile or pe-list directives)</p>
<div class="admonition error">
<p class="admonition-title">Error</p>
<p>JMS Several of the options below refer to <code class="docutils literal notranslate"><span class="pre">pe-list</span></code>.
Is this option supposed to be <code class="docutils literal notranslate"><span class="pre">PE-LIST=n</span></code>, not
<code class="docutils literal notranslate"><span class="pre">PE=n</span></code>?</p>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">SPAN</span></code> load balance the processes across the allocation (cannot
be used in combination with <code class="docutils literal notranslate"><span class="pre">slot</span></code>, <code class="docutils literal notranslate"><span class="pre">node</span></code>, <code class="docutils literal notranslate"><span class="pre">seq</span></code>, <code class="docutils literal notranslate"><span class="pre">ppr</span></code>,
<code class="docutils literal notranslate"><span class="pre">rankfile</span></code>, or <code class="docutils literal notranslate"><span class="pre">pe-list</span></code> directives)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">OVERSUBSCRIBE</span></code> allow more processes on a node than processing
elements</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">NOOVERSUBSCRIBE</span></code> means <code class="docutils literal notranslate"><span class="pre">!OVERSUBSCRIBE</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">NOLOCAL</span></code> do not launch processes on the same node as <code class="docutils literal notranslate"><span class="pre">prun</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">HWTCPUS</span></code> use hardware threads as CPU slots</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">CORECPUS</span></code> use cores as CPU slots (default)</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">INHERIT</span></code> indicates that a child job (i.e., one spawned from
within an application) shall inherit the placement policies of the
parent job that spawned it.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">NOINHERIT</span></code> means <code class="docutils literal notranslate"><span class="pre">!INHERIT</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">FILE=&lt;path&gt;</span></code> (path to file containing sequential or rankfile
entries).</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ORDERED</span></code> only applies to the PE-LIST option to indicate that
procs are to be bound to each of the specified CPUs in the order
in which they are assigned (i.e., the first proc on a node shall
be bound to the first CPU in the list, the second proc shall be
bound to the second CPU, etc.)</p></li>
</ul>
<p><code class="docutils literal notranslate"><span class="pre">ppr</span></code> policy example: <code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">ppr:N:&lt;object&gt;</span></code> will launch
<code class="docutils literal notranslate"><span class="pre">N</span></code> times the number of objects of the specified type on each
node.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Directives and qualifiers are case-insensitive and can be
shortened to the minimum number of characters to uniquely
identify them. Thus, <code class="docutils literal notranslate"><span class="pre">L1CACHE</span></code> can be given as
<code class="docutils literal notranslate"><span class="pre">l1cache</span></code> or simply as <code class="docutils literal notranslate"><span class="pre">L1</span></code>.</p>
</div>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">--rank-by</span> <span class="pre">&lt;object&gt;</span></code>: This assigns ranks in round-robin fashion
according to the specified object. The default follows the mapping
pattern. Supported rank-by objects include:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">slot</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">node</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">fill</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">span</span></code></p></li>
</ul>
<p>There are no qualifiers for the <code class="docutils literal notranslate"><span class="pre">--rank-by</span></code> directive.</p>
</li>
<li><p><code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">&lt;object&gt;</span></code>: This binds processes to the specified
object. See defaults in Quick Summary. Supported bind-to objects
include:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">none</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">hwthread</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">core</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">l1cache</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">l2cache</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">l3cache</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">numa</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">package</span></code></p></li>
</ul>
<p>Any object can include qualifiers by adding a colon (<code class="docutils literal notranslate"><span class="pre">:</span></code>) and any
colon-delimited combination of one or more of the following to the
<code class="docutils literal notranslate"><span class="pre">--bind-to</span></code> options:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">overload-allowed</span></code> allows for binding more than one process in
relation to a CPU</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">if-supported</span></code> if binding to that object is supported on this
system.</p></li>
</ul>
</li>
</ul>
</div>
<div class="section" id="id1">
<h2><span class="section-number">8.4.3. </span>Specifying Host Nodes<a class="headerlink" href="#id1" title="Permalink to this heading"></a></h2>
<p>Host nodes can be identified on the command line with the <code class="docutils literal notranslate"><span class="pre">--host</span></code>
option or in a hostfile.</p>
<p>For example, assuming no other resource manager or scheduler is
involved:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">host</span> <span class="n">aa</span><span class="p">,</span><span class="n">aa</span><span class="p">,</span><span class="n">bb</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This launches two processes on node <code class="docutils literal notranslate"><span class="pre">aa</span></code> and one on <code class="docutils literal notranslate"><span class="pre">bb</span></code>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">host</span> <span class="n">aa</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This launches one process on node <code class="docutils literal notranslate"><span class="pre">aa</span></code>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">host</span> <span class="n">aa</span><span class="p">:</span><span class="mi">5</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This launches five processes on node <code class="docutils literal notranslate"><span class="pre">aa</span></code>.</p>
<p>Or, consider the hostfile:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
aa slots=2
bb slots=2
cc slots=2
</pre></div>
</div>
<p>Here, we list both the host names (<code class="docutils literal notranslate"><span class="pre">aa</span></code>, <code class="docutils literal notranslate"><span class="pre">bb</span></code>, and <code class="docutils literal notranslate"><span class="pre">cc</span></code>) but
also how many “slots” there are for each. Slots indicate how many
processes can potentially execute on a node. For best performance, the
number of slots may be chosen to be the number of cores on the node or
the number of processor sockets.</p>
<p>If the hostfile does not provide slots information, the PRRTE DVM will
attempt to discover the number of cores (or hwthreads, if the
<code class="docutils literal notranslate"><span class="pre">:HWTCPUS</span></code> qualifier to the <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> option is set) and set the
number of slots to that value.</p>
<p>Examples using the hostfile above with and without the <code class="docutils literal notranslate"><span class="pre">--host</span></code>
option:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This will launch two processes on each of the three nodes.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="n">host</span> <span class="n">aa</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This will launch two processes, both on node <code class="docutils literal notranslate"><span class="pre">aa</span></code>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="n">host</span> <span class="n">dd</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This will find no hosts to run on and abort with an error. That is, the
specified host <code class="docutils literal notranslate"><span class="pre">dd</span></code> is not in the specified hostfile.</p>
<p>When running under resource managers (e.g., SLURM, Torque, etc.), PRTE
will obtain both the hostnames and the number of slots directly from
the resource manger. The behavior of <code class="docutils literal notranslate"><span class="pre">--host</span></code> in that environment
will behave the same as if a hostfile was provided (since it is
provided by the resource manager).</p>
</div>
<div class="section" id="specifying-number-of-processes">
<h2><span class="section-number">8.4.4. </span>Specifying Number of Processes<a class="headerlink" href="#specifying-number-of-processes" title="Permalink to this heading"></a></h2>
<p>As we have just seen, the number of processes to run can be set using
the hostfile. Other mechanisms exist.</p>
<p>The number of processes launched can be specified as a multiple of the
number of nodes or processor sockets available. Consider the hostfile
below for the examples that follow.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
aa
bb
</pre></div>
</div>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">ppr</span><span class="p">:</span><span class="mi">2</span><span class="p">:</span><span class="n">package</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This launches processes 0-3 on node <code class="docutils literal notranslate"><span class="pre">aa</span></code> and process 4-7 on node
<code class="docutils literal notranslate"><span class="pre">bb</span></code>, where <code class="docutils literal notranslate"><span class="pre">aa</span></code> and <code class="docutils literal notranslate"><span class="pre">bb</span></code> are both dual-package nodes. The
<code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">ppr:2:package</span></code> option also turns on the <code class="docutils literal notranslate"><span class="pre">--bind-to</span>
<span class="pre">package</span></code> option, which is discussed in a later section.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">ppr</span><span class="p">:</span><span class="mi">2</span><span class="p">:</span><span class="n">node</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This launches processes 0-1 on node <code class="docutils literal notranslate"><span class="pre">aa</span></code> and processes 2-3 on node
<code class="docutils literal notranslate"><span class="pre">bb</span></code>.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">ppr</span><span class="p">:</span><span class="mi">1</span><span class="p">:</span><span class="n">node</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This launches one process per host node.</p>
<p>Another alternative is to specify the number of processes with the
<code class="docutils literal notranslate"><span class="pre">--np</span></code> option. Consider now the hostfile:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
aa slots=4
bb slots=4
cc slots=4
</pre></div>
</div>
<p>With this hostfile:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="n">np</span> <span class="mi">6</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This will launch processes 0-3 on node <code class="docutils literal notranslate"><span class="pre">aa</span></code> and processes 4-5 on
node <code class="docutils literal notranslate"><span class="pre">bb</span></code>. The remaining slots in the hostfile will not be used
since the <code class="docutils literal notranslate"><span class="pre">-np</span></code> option indicated that only 6 processes should be
launched.</p>
</div>
<div class="section" id="mapping-processes-to-nodes-using-policies">
<h2><span class="section-number">8.4.5. </span>Mapping Processes to Nodes Using Policies<a class="headerlink" href="#mapping-processes-to-nodes-using-policies" title="Permalink to this heading"></a></h2>
<p>The examples above illustrate the default mapping of process processes
to nodes. This mapping can also be controlled with various
<code class="docutils literal notranslate"><span class="pre">prun</span></code> / <code class="docutils literal notranslate"><span class="pre">prterun</span></code> options that describe mapping policies.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
aa slots=4
bb slots=4
cc slots=4
</pre></div>
</div>
<p>Consider the hostfile above, with <code class="docutils literal notranslate"><span class="pre">--np</span> <span class="pre">6</span></code>:</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 25%" />
<col style="width: 25%" />
<col style="width: 25%" />
<col style="width: 25%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Command</p></th>
<th class="head"><p>Ranks on <code class="docutils literal notranslate"><span class="pre">aa</span></code></p></th>
<th class="head"><p>Ranks on <code class="docutils literal notranslate"><span class="pre">bb</span></code></p></th>
<th class="head"><p>Ranks on <code class="docutils literal notranslate"><span class="pre">cc</span></code></p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">prun</span></code></p></td>
<td><p>0 1 2 3</p></td>
<td><p>4 5</p></td>
<td></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">prun</span> <span class="pre">--map-by</span> <span class="pre">node</span></code></p></td>
<td><p>0 3</p></td>
<td><p>1 4</p></td>
<td><p>2 5</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">prun</span> <span class="pre">--map-by</span> <span class="pre">node:NOLOCAL</span></code></p></td>
<td></td>
<td><p>0 2 4</p></td>
<td><p>1 3 5</p></td>
</tr>
</tbody>
</table>
<p>The <code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">node</span></code> option will load balance the processes across
the available nodes, numbering each process by node in a round-robin
fashion.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">:NOLOCAL</span></code> qualifier to <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> prevents any processes from
being mapped onto the local host (in this case node <code class="docutils literal notranslate"><span class="pre">aa</span></code>). While
<code class="docutils literal notranslate"><span class="pre">prun</span></code> typically consumes few system resources, the <code class="docutils literal notranslate"><span class="pre">:NOLOCAL</span></code>
qualifier can be helpful for launching very large jobs where <code class="docutils literal notranslate"><span class="pre">prun</span></code>
may actually need to use noticeable amounts of memory and/or
processing time.</p>
<p>Just as <code class="docutils literal notranslate"><span class="pre">--np</span></code> can specify fewer processes than there are slots, it
can also oversubscribe the slots. For example, with the same hostfile:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="n">np</span> <span class="mi">14</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This will produce an error since the default <code class="docutils literal notranslate"><span class="pre">:NOOVERSUBSCRIBE</span></code>
qualifier to <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> prevents oversubscription.</p>
<p>To oversubscribe the nodes you can use the <code class="docutils literal notranslate"><span class="pre">:OVERSUBSCRIBE</span></code>
qualifier to <code class="docutils literal notranslate"><span class="pre">--map-by</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="n">np</span> <span class="mi">14</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="p">:</span><span class="n">OVERSUBSCRIBE</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This will launch processes 0-5 on node <code class="docutils literal notranslate"><span class="pre">aa</span></code>, 6-9 on <code class="docutils literal notranslate"><span class="pre">bb</span></code>, and
10-13 on <code class="docutils literal notranslate"><span class="pre">cc</span></code>.</p>
<p>Limits to oversubscription can also be specified in the hostfile
itself with the <code class="docutils literal notranslate"><span class="pre">max_slots</span></code> field:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
aa slots=4 max_slots=4
bb max_slots=8
cc slots=4
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">max_slots</span></code> field specifies such a limit. When it does, the
<code class="docutils literal notranslate"><span class="pre">slots</span></code> value defaults to the limit. Now:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="n">np</span> <span class="mi">14</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="p">:</span><span class="n">OVERSUBSCRIBE</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This causes the first 12 processes to be launched as before, but the
remaining two processes will be forced onto node cc. The other two
nodes are protected by the hostfile against oversubscription by this
job.</p>
<p>Using the <code class="docutils literal notranslate"><span class="pre">:NOOVERSUBSCRIBE</span></code> qualifier to <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> option can be
helpful since the PRTE DVM currently does not get <code class="docutils literal notranslate"><span class="pre">max_slots</span></code> values
from the resource manager.</p>
<p>Of course, <code class="docutils literal notranslate"><span class="pre">--np</span></code> can also be used with the <code class="docutils literal notranslate"><span class="pre">--host</span></code> option. For
example,</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">host</span> <span class="n">aa</span><span class="p">,</span><span class="n">bb</span> <span class="o">--</span><span class="n">np</span> <span class="mi">8</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This will produce an error since the default <code class="docutils literal notranslate"><span class="pre">:NOOVERSUBSCRIBE</span></code>
qualifier to <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> prevents oversubscription.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">host</span> <span class="n">aa</span><span class="p">,</span><span class="n">bb</span> <span class="o">--</span><span class="n">np</span> <span class="mi">8</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="p">:</span><span class="n">OVERSUBSCRIBE</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This launches 8 processes. Since only two hosts are specified, after
the first two processes are mapped, one to <code class="docutils literal notranslate"><span class="pre">aa</span></code> and one to <code class="docutils literal notranslate"><span class="pre">bb</span></code>,
the remaining processes oversubscribe the specified hosts evenly.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">host</span> <span class="n">aa</span><span class="p">:</span><span class="mi">2</span><span class="p">,</span><span class="n">bb</span><span class="p">:</span><span class="mi">6</span> <span class="o">--</span><span class="n">np</span> <span class="mi">8</span> <span class="o">./</span><span class="n">a</span><span class="o">.</span><span class="n">out</span>
</pre></div>
</div>
<p>This launches 8 processes. Processes 0-1 on node <code class="docutils literal notranslate"><span class="pre">aa</span></code> since it has 2
slots and processes 2-7 on node <code class="docutils literal notranslate"><span class="pre">bb</span></code> since it has 6 slots.</p>
<p>And here is a MIMD example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">host</span> <span class="n">aa</span> <span class="o">--</span><span class="n">np</span> <span class="mi">1</span> <span class="n">hostname</span> <span class="p">:</span> <span class="o">--</span><span class="n">host</span> <span class="n">bb</span><span class="p">,</span><span class="n">cc</span> <span class="o">--</span><span class="n">np</span> <span class="mi">2</span> <span class="n">uptime</span>
</pre></div>
</div>
<p>This will launch process 0 running <code class="docutils literal notranslate"><span class="pre">hostname</span></code> on node <code class="docutils literal notranslate"><span class="pre">aa</span></code> and
processes 1 and 2 each running <code class="docutils literal notranslate"><span class="pre">uptime</span></code> on nodes <code class="docutils literal notranslate"><span class="pre">bb</span></code> and <code class="docutils literal notranslate"><span class="pre">cc</span></code>,
respectively.</p>
</div>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="overview.html" class="btn btn-neutral float-left" title="8.1. Overview" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="fundamentals.html" class="btn btn-neutral float-right" title="8.5. Fundamentals" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>

View File

@@ -0,0 +1,281 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8.5. Fundamentals &mdash; PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="8.6. Overloading and Oversubscribing" href="limits.html" />
<link rel="prev" title="8.4. Examples" href="examples.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">8. Process placement</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="overview.html">8.1. Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-slot">8.2. Definition of ‘slot’</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">8.5. Fundamentals</a></li>
<li class="toctree-l2"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l2"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l2"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">8. </span>Process placement</a></li>
<li class="breadcrumb-item active"><span class="section-number">8.5. </span>Fundamentals</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/fundamentals.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="fundamentals">
<h1><span class="section-number">8.5. </span>Fundamentals<a class="headerlink" href="#fundamentals" title="Permalink to this heading"></a></h1>
<p>The mapping of processes to nodes can be defined not just with general
policies but also, if necessary, using arbitrary mappings that cannot
be described by a simple policy. Supported directives, given on the
command line via the <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> option, include:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">SEQ</span></code>: (often accompanied by the <code class="docutils literal notranslate"><span class="pre">file=&lt;path&gt;</span></code> qualifier)
assigns one process to each node specified in the file. The
sequential file is to contain an entry for each desired process, one
per line of the file.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">RANKFILE</span></code>: (often accompanied by the <code class="docutils literal notranslate"><span class="pre">file=&lt;path&gt;</span></code> qualifier)
assigns one process to the node/resource specified in each entry of
the file, one per line of the file.</p></li>
</ul>
<p>For example, using the hostfile below:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
aa slots=4
bb slots=4
cc slots=4
</pre></div>
</div>
<p>The command below will launch three processes, one on each of nodes
<code class="docutils literal notranslate"><span class="pre">aa</span></code>, <code class="docutils literal notranslate"><span class="pre">bb</span></code>, and <code class="docutils literal notranslate"><span class="pre">cc</span></code>, respectively. The slot counts don’t
matter; one process is launched per line on whatever node is listed on
the line.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ prun --hostfile myhostfile --map-by seq ./a.out
</pre></div>
</div>
<p>Impact of the ranking option is best illustrated by considering the
following hostfile and test cases where each node contains two
packages (each package with two cores). Using the <code class="docutils literal notranslate"><span class="pre">--map-by</span>
<span class="pre">ppr:2:package</span></code> option, we map two processes onto each package and
utilize the <code class="docutils literal notranslate"><span class="pre">--rank-by</span></code> option as show below:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
aa
bb
</pre></div>
</div>
<table class="docutils align-default">
<colgroup>
<col style="width: 33%" />
<col style="width: 33%" />
<col style="width: 33%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Command</p></th>
<th class="head"><p>Ranks on <code class="docutils literal notranslate"><span class="pre">aa</span></code></p></th>
<th class="head"><p>Ranks on <code class="docutils literal notranslate"><span class="pre">bb</span></code></p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--rank-by</span> <span class="pre">core</span></code></p></td>
<td><p>0 1 ! 2 3</p></td>
<td><p>4 5 ! 6 7</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--rank-by</span> <span class="pre">package</span></code></p></td>
<td><p>0 2 ! 1 3</p></td>
<td><p>4 6 ! 5 7</p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--rank-by</span> <span class="pre">package:SPAN</span></code></p></td>
<td><p>0 4 ! 1 5</p></td>
<td><p>2 6 ! 3 7</p></td>
</tr>
</tbody>
</table>
<p>Ranking by slot provides the identical result as ranking by core in
this case — a simple progression of ranks across each
node. Ranking by package does a round-robin ranking across packages
within each node until all processes have been assigned a rank, and
then progresses to the next node. Adding the <code class="docutils literal notranslate"><span class="pre">:SPAN</span></code> qualifier to
the ranking directive causes the ranking algorithm to treat the entire
allocation as a single entity — thus, the process ranks are
assigned across all packages before circling back around to the
beginning.</p>
<p>The binding operation restricts the process to a subset of the CPU
resources on the node.</p>
<p>The processors to be used for binding can be identified in terms of
topological groupings — e.g., binding to an l3cache will bind
each process to all processors within the scope of a single L3 cache
within their assigned location. Thus, if a process is assigned by the
mapper to a certain package, then a <code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">l3cache</span></code> directive
will cause the process to be bound to the processors that share a
single L3 cache within that package.</p>
<p>To help balance loads, the binding directive uses a round-robin method,
binding a process to the first available specified object type within
the object where the process was mapped. For example, consider the case
where a job is mapped to the package level, and then bound to core. Each
package will have multiple cores, so if multiple processes are mapped to
a given package, the binding algorithm will assign each process located
to a package to a unique core in a round-robin manner.</p>
<p>Binding can only be done to the mapped object or to a resource located
within that object.</p>
<p>An object is considered completely consumed when the number of
processes bound to it equals the number of CPUs within it. Unbound
processes are not considered in this computation. Additional
processes cannot be mapped to consumed objects unless the
OVERLOAD qualifier is provided via the “–bind-to” command
line option.</p>
<p>Default process mapping/ranking/binding policies can also be set with MCA
parameters, overridden by the command line options when provided. MCA
parameters can be set on the <code class="docutils literal notranslate"><span class="pre">prte</span></code> command line when starting the
DVM (or in the <code class="docutils literal notranslate"><span class="pre">prterun</span></code> command line for a single-execution job), but
also in a system or user <code class="docutils literal notranslate"><span class="pre">mca-params.conf</span></code> file or as environment
variables, as described in the MCA section below. Some examples include:</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 33%" />
<col style="width: 33%" />
<col style="width: 33%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p><code class="docutils literal notranslate"><span class="pre">prun</span></code> option</p></th>
<th class="head"><p>MCA parameter key</p></th>
<th class="head"><p>Value</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">core</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">rmaps_default_mapping_policy</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">core</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">package</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">rmaps_default_mapping_policy</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">package</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--rank-by</span> <span class="pre">core</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">rmaps_default_ranking_policy</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">core</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">core</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">hwloc_default_binding_policy</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">core`</span></code></p></td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">package</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">hwloc_default_binding_policy</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">package</span></code></p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">none</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">hwloc_default_binding_policy</span></code></p></td>
<td><p><code class="docutils literal notranslate"><span class="pre">none</span></code></p></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="examples.html" class="btn btn-neutral float-left" title="8.4. Examples" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="limits.html" class="btn btn-neutral float-right" title="8.6. Overloading and Oversubscribing" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>

View File

@@ -0,0 +1,159 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8. Process placement &mdash; PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="8.1. Overview" href="overview.html" />
<link rel="prev" title="7.4. Relative host indexing" href="../hosts/relative-indexing.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">8. Process placement</a><ul>
<li class="toctree-l2"><a class="reference internal" href="overview.html">8.1. Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-slot">8.2. Definition of ‘slot’</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l2"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l2"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l2"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item active"><span class="section-number">8. </span>Process placement</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/index.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="process-placement">
<h1><span class="section-number">8. </span>Process placement<a class="headerlink" href="#process-placement" title="Permalink to this heading"></a></h1>
<p>PRRTE provides a set of three controls for assigning process
locations and ranks:</p>
<ol class="arabic simple">
<li><p>Mapping: Assigns a default location to each process</p></li>
<li><p>Ranking: Assigns a unique integer rank value to each process</p></li>
<li><p>Binding: Constrains each process to run on specific processors</p></li>
</ol>
<p>The sections below describe these controls in more detail.</p>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="overview.html">8.1. Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="overview.html#definition-of-slot">8.2. Definition of ‘slot’</a></li>
<li class="toctree-l1"><a class="reference internal" href="overview.html#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l1"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l1"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l1"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l1"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l1"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l1"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</div>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="../hosts/relative-indexing.html" class="btn btn-neutral float-left" title="7.4. Relative host indexing" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="overview.html" class="btn btn-neutral float-right" title="8.1. Overview" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>

View File

@@ -0,0 +1,295 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8.6. Overloading and Oversubscribing &mdash; PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="8.7. Diagnostics" href="diagnostics.html" />
<link rel="prev" title="8.5. Fundamentals" href="fundamentals.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">8. Process placement</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="overview.html">8.1. Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-slot">8.2. Definition of ‘slot’</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">8.6. Overloading and Oversubscribing</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#overloading-vs-oversubscription-package-example">8.6.1. Overloading vs. Oversubscription: Package Example</a></li>
<li class="toctree-l3"><a class="reference internal" href="#overloading-vs-oversubscription-hardware-threads-example">8.6.2. Overloading vs. Oversubscription: Hardware Threads Example</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l2"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">8. </span>Process placement</a></li>
<li class="breadcrumb-item active"><span class="section-number">8.6. </span>Overloading and Oversubscribing</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/limits.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="overloading-and-oversubscribing">
<h1><span class="section-number">8.6. </span>Overloading and Oversubscribing<a class="headerlink" href="#overloading-and-oversubscribing" title="Permalink to this heading"></a></h1>
<p>This section explores the difference between the terms “overloading”
and “oversubscribing”. Users are often confused by the difference
between these two scenarios. As such, this section provides a number
of scenarios to help illustrate the differences.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">--map-by</span> <span class="pre">:OVERSUBSCRIBE</span></code> allow more processes on a node than
allocated</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">&lt;object&gt;:overload-allowed</span></code> allows for binding more than
one process in relation to a CPU</p></li>
</ul>
<p>The important thing to remember with <em>oversubscribing</em> is that it can
be defined separately from the actual number of CPUs on a node. This
allows the mapper to place more or fewer processes per node than
CPUs. By default, PRRTE uses cores to determine slots in the absence
of such information provided in the hostfile or by the resource
manager (except in the case of the <code class="docutils literal notranslate"><span class="pre">--host</span></code> as described in
the section on that command line option.</p>
<p>The important thing to remember with <em>overloading</em> is that it is
defined as binding more processes than CPUs. By default, PRRTE uses
cores as a means of counting the number of CPUs. However, the user can
adjust this. For example when using the <code class="docutils literal notranslate"><span class="pre">:HWTCPUS</span></code> qualifier to the
<code class="docutils literal notranslate"><span class="pre">--map-by</span></code> option PRRTE will use hardware threads as a means of
counting the number of CPUs.</p>
<p>For the following examples consider a node with:</p>
<ul class="simple">
<li><p>2 processor packages,</p></li>
<li><p>10 cores per package, and</p></li>
<li><p>8 hardware threads per core.</p></li>
</ul>
<p>Consider the node from above with the hostfile below:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
node01 slots=32
node02 slots=32
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">slots</span></code> token tells PRRTE that it can place up to 32 processes
before <em>oversubscribing</em> the node.</p>
<p>If we run the following:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">34</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">core</span> <span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">core</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>It will return an error at the binding time indicating an
<em>overloading</em> scenario.</p>
<p>The mapping mechanism assigns 32 processes to <code class="docutils literal notranslate"><span class="pre">node01</span></code> matching the
<code class="docutils literal notranslate"><span class="pre">slots</span></code> specification in the hostfile. The binding mechanism will bind
the first 20 processes to unique cores leaving it with 12 processes
that it cannot bind without overloading one of the cores (putting more
than one process on the core).</p>
<p>Using the <code class="docutils literal notranslate"><span class="pre">overload-allowed</span></code> qualifier to the <code class="docutils literal notranslate"><span class="pre">--bind-to</span> <span class="pre">core</span></code>
option tells PRRTE that it may assign more than one process to a core.</p>
<p>If we run the following:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">34</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">core</span> <span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">core</span><span class="p">:</span><span class="n">overload</span><span class="o">-</span><span class="n">allowed</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>This will run correctly placing 32 processes on <code class="docutils literal notranslate"><span class="pre">node01</span></code>, and 2
processes on <code class="docutils literal notranslate"><span class="pre">node02</span></code>. On <code class="docutils literal notranslate"><span class="pre">node01</span></code> two processes are bound to
cores 0-11 accounting for the overloading of those cores.</p>
<p>Alternatively, we could use hardware threads to give binding a lower
level CPU to bind to without overloading.</p>
<p>If we run the following:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">34</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">core</span><span class="p">:</span><span class="n">HWTCPUS</span> <span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">hwthread</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>This will run correctly placing 32 processes on <code class="docutils literal notranslate"><span class="pre">node01</span></code>, and 2
processes on <code class="docutils literal notranslate"><span class="pre">node02</span></code>. On <code class="docutils literal notranslate"><span class="pre">node01</span></code> two processes are mapped to
cores 0-11 but bound to different hardware threads on those cores (the
logical first and second hardware thread). Thus no hardware threads
are overloaded at binding time.</p>
<p>In both of the examples above the node is not oversubscribed at
mapping time because the hostfile set the oversubscription limit to
<code class="docutils literal notranslate"><span class="pre">slots=32</span></code> for each node. It is only after we exceed that limit that
PRRTE will throw an oversubscription error.</p>
<p>Consider next if we ran the following:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">66</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">core</span><span class="p">:</span><span class="n">HWTCPUS</span> <span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">hwthread</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>This will return an error at mapping time indicating an
oversubscription scenario. The mapping mechanism will assign all of
the available slots (64 across 2 nodes) and be left two processes to
map. The only way to map those processes is to exceed the number of
available slots putting the job into an oversubscription scenario.</p>
<p>You can force PRRTE to oversubscribe the nodes by using the
<code class="docutils literal notranslate"><span class="pre">:OVERSUBSCRIBE</span></code> qualifier to the <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> option as seen in the
example below:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">66</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> \
<span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">core</span><span class="p">:</span><span class="n">HWTCPUS</span><span class="p">:</span><span class="n">OVERSUBSCRIBE</span> <span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">hwthread</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>This will run correctly placing 34 processes on <code class="docutils literal notranslate"><span class="pre">node01</span></code> and 32 on
<code class="docutils literal notranslate"><span class="pre">node02</span></code>. Each process is bound to a unique hardware thread.</p>
<div class="section" id="overloading-vs-oversubscription-package-example">
<h2><span class="section-number">8.6.1. </span>Overloading vs. Oversubscription: Package Example<a class="headerlink" href="#overloading-vs-oversubscription-package-example" title="Permalink to this heading"></a></h2>
<p>Let’s extend these examples by considering the package level.
Consider the same node as before, but with the hostfile below:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
node01 slots=22
node02 slots=22
</pre></div>
</div>
<p>The lowest level CPUs are “cores” and we have 20 total (10 per
package).</p>
<p>If we run:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">20</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">package</span> \
<span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">package</span><span class="p">:</span><span class="n">REPORT</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>Then 10 processes are mapped to each package, and bound at the package
level. This is not overloading since we have 10 CPUs (cores)
available in the package at the hardware level.</p>
<p>However, if we run:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">21</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">package</span> \
<span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">package</span><span class="p">:</span><span class="n">REPORT</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>Then 11 processes are mapped to the first package and 10 to the second
package. At binding time we have an overloading scenario because
there are only 10 CPUs (cores) available in the package at the
hardware level. So the first package is overloaded.</p>
</div>
<div class="section" id="overloading-vs-oversubscription-hardware-threads-example">
<h2><span class="section-number">8.6.2. </span>Overloading vs. Oversubscription: Hardware Threads Example<a class="headerlink" href="#overloading-vs-oversubscription-hardware-threads-example" title="Permalink to this heading"></a></h2>
<p>Similarly, if we consider hardware threads.</p>
<p>Consider the same node as before, but with the hostfile below:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myhostfile
node01 slots=165
node02 slots=165
</pre></div>
</div>
<p>The lowest level CPUs are “hwthreads” (because we are going to use the
<code class="docutils literal notranslate"><span class="pre">:HWTCPUS</span></code> qualifier) and we have 160 total (80 per package).</p>
<p>If we re-run (from the package example) and add the <code class="docutils literal notranslate"><span class="pre">:HWTCPUS</span></code>
qualifier:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">21</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">package</span><span class="p">:</span><span class="n">HWTCPUS</span> \
<span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">package</span><span class="p">:</span><span class="n">REPORT</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>Without the <code class="docutils literal notranslate"><span class="pre">:HWTCPUS</span></code> qualifier this would be overloading (as we
saw previously). The mapper places 11 processes on the first package
and 10 to the second package. The processes are still bound to the
package level. However, with the <code class="docutils literal notranslate"><span class="pre">:HWTCPUS</span></code> qualifier, it is not
overloading since we have 80 CPUs (hwthreads) available in the package
at the hardware level.</p>
<p>Alternatively, if we run:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">prun</span> <span class="o">--</span><span class="n">np</span> <span class="mi">161</span> <span class="o">--</span><span class="n">hostfile</span> <span class="n">myhostfile</span> <span class="o">--</span><span class="nb">map</span><span class="o">-</span><span class="n">by</span> <span class="n">package</span><span class="p">:</span><span class="n">HWTCPUS</span> \
<span class="o">--</span><span class="n">bind</span><span class="o">-</span><span class="n">to</span> <span class="n">package</span><span class="p">:</span><span class="n">REPORT</span> <span class="n">hostname</span>
</pre></div>
</div>
<p>Then 81 processes are mapped to the first package and 80 to the second
package. At binding time we have an overloading scenario because
there are only 80 CPUs (hwthreads) available in the package at the
hardware level. So the first package is overloaded.</p>
</div>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="fundamentals.html" class="btn btn-neutral float-left" title="8.5. Fundamentals" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="diagnostics.html" class="btn btn-neutral float-right" title="8.7. Diagnostics" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>

View File

@@ -0,0 +1,300 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8.1. Overview &mdash; PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="8.4. Examples" href="examples.html" />
<link rel="prev" title="8. Process placement" href="index.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">8. Process placement</a><ul class="current">
<li class="toctree-l2 current"><a class="current reference internal" href="#">8.1. Overview</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#quick-summary">8.1.1. Quick Summary</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#definition-of-slot">8.2. Definition of ‘slot’</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#slots-are-not-hardware-resources">8.2.1. Slots are not hardware resources</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l2"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l2"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l2"><a class="reference internal" href="rankfiles.html">8.8. Rankfiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">8. </span>Process placement</a></li>
<li class="breadcrumb-item active"><span class="section-number">8.1. </span>Overview</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/overview.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="overview">
<h1><span class="section-number">8.1. </span>Overview<a class="headerlink" href="#overview" title="Permalink to this heading"></a></h1>
<p>PRRTE provides a set of three controls for assigning process
locations and ranks:</p>
<ol class="arabic simple">
<li><p>Mapping: Assigns a default location to each process</p></li>
<li><p>Ranking: Assigns a unique integer rank value to each process</p></li>
<li><p>Binding: Constrains each process to run on specific processors</p></li>
</ol>
<p>This section provides an overview of these three controls. Unless
otherwise this behavior is shared by <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code> (working with a PRRTE
DVM), and <code class="docutils literal notranslate"><span class="pre">prterun(1)</span></code>. More detail about PRRTE process placement is
available in the following sections (using <code class="docutils literal notranslate"><span class="pre">--help</span>
<span class="pre">placement-&lt;section&gt;</span></code>):</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">examples</span></code>: some examples of the interactions between mapping,
ranking, and binding options.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">fundamentals</span></code>: provides deeper insight into PRRTE’s mapping,
ranking, and binding options.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">limits</span></code>: explains the difference between <em>overloading</em> and
<em>oversubscribing</em> resources.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">diagnostics</span></code>: describes options for obtaining various diagnostic
reports that aid the user in verifying and tuning the placement for
a specific job.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">rankfiles</span></code>: explains the format and use of the rankfile mapper
for specifying arbitrary process placements.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">deprecated</span></code>: a list of deprecated options and their new
equivalents.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">all</span></code>: outputs all the placement help except for the
<code class="docutils literal notranslate"><span class="pre">deprecated</span></code> section.</p></li>
</ul>
<div class="section" id="quick-summary">
<h2><span class="section-number">8.1.1. </span>Quick Summary<a class="headerlink" href="#quick-summary" title="Permalink to this heading"></a></h2>
<p>The two binaries that most influence process layout are <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code>
and <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code>. The <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> process discovers the allocation,
establishes a Distributed Virtual Machine by starting a <code class="docutils literal notranslate"><span class="pre">prted(1)</span></code>
daemon on each node of the allocation, and defines the efault
mapping/ranking/binding policies for all jobs. The <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code> process
defines the specific mapping/ranking/binding for a specific job. Most
of the command line controls are targeted to <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code> since each job
has its own unique requirements.</p>
<p><code class="docutils literal notranslate"><span class="pre">prterun(1)</span></code> is just a wrapper around <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> for a single job
PRRTE DVM. It is doing the job of both <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> and <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code>,
and, as such, accepts the sum all of their command line arguments. Any
example that uses <code class="docutils literal notranslate"><span class="pre">prun(1)</span></code> can substitute the use of <code class="docutils literal notranslate"><span class="pre">prterun(1)</span></code>
except where otherwise noted.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> process attempts to automatically discover the nodes
in the allocation by querying supported resource managers. If a
supported resource manager is not present then <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> relies on a
hostfile provided by the user. In the absence of such a hostfile it
will run all processes on the localhost.</p>
<p>If running under a supported resource manager, the <code class="docutils literal notranslate"><span class="pre">prte(1)</span></code> process
will start the daemon processes (<code class="docutils literal notranslate"><span class="pre">prted(1)</span></code>) on the remote nodes
using the corresponding resource manager process starter. If no such
starter is available then <code class="docutils literal notranslate"><span class="pre">ssh</span></code> (or <code class="docutils literal notranslate"><span class="pre">rsh</span></code>) is used.</p>
<p>Minus user direction, PRRTE will automatically map processes in a
round-robin fashion by CPU, binding each process to its own CPU. The
type of CPU used (core vs hwthread) is determined by (in priority
order):</p>
<ul class="simple">
<li><p>user directive on the command line via the HWTCPUS qualifier to
the <code class="docutils literal notranslate"><span class="pre">--map-by</span></code> directive</p></li>
<li><p>setting the <code class="docutils literal notranslate"><span class="pre">rmaps_default_mapping_policy</span></code> MCA parameter to
include the <code class="docutils literal notranslate"><span class="pre">HWTCPUS</span></code> qualifier. This parameter sets the default
value for a PRRTE DVM — qualifiers are carried across to DVM
jobs started via <code class="docutils literal notranslate"><span class="pre">prun</span></code> unless overridden by the user’s command
line</p></li>
<li><p>defaulting to <code class="docutils literal notranslate"><span class="pre">CORE</span></code> in topologies where core CPUs are defined,
and to <code class="docutils literal notranslate"><span class="pre">hwthreads</span></code> otherwise.</p></li>
</ul>
<p>By default, the ranks are assigned in accordance with the mapping
directive — e.g., jobs that are mapped by-node will have the
process ranks assigned round-robin on a per-node basis.</p>
<p>PRRTE automatically binds processes unless directed not to do so by
the user. Minus direction, PRRTE will bind individual processes to
their own CPU within the object to which they were mapped. Should a
node become oversubscribed during the mapping process, and if
oversubscription is allowed, all subsequent processes assigned to that
node will <em>not</em> be bound.</p>
</div>
</div>
<div class="section" id="definition-of-slot">
<h1><span class="section-number">8.2. </span>Definition of ‘slot’<a class="headerlink" href="#definition-of-slot" title="Permalink to this heading"></a></h1>
<p>The term “slot” is used extensively in the rest of this documentation.
A slot is an allocation unit for a process. The number of slots on a
node indicate how many processes can potentially execute on that node.
By default, PRRTE will allow one process per slot.</p>
<p>If PRRTE is not explicitly told how many slots are available on a node
(e.g., if a hostfile is used and the number of slots is not specified
for a given node), it will determine a maximum number of slots for
that node in one of two ways:</p>
<ol class="arabic simple">
<li><p>Default behavior: By default, PRRTE will attempt to discover the
number of processor cores on the node, and use that as the number
of slots available.</p></li>
<li><p>When <code class="docutils literal notranslate"><span class="pre">--use-hwthread-cpus</span></code> is used: If <code class="docutils literal notranslate"><span class="pre">--use-hwthread-cpus</span></code> is
specified on the command line, then PRRTE will attempt to discover
the number of hardware threads on the node, and use that as the
number of slots available.</p></li>
</ol>
<p>This default behavior also occurs when specifying the <code class="docutils literal notranslate"><span class="pre">--host</span></code>
option with a single host. Thus, the command:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>prun<span class="w"> </span>--host<span class="w"> </span>node1<span class="w"> </span>./a.out
</pre></div>
</div>
<p>launches a number of processes equal to the number of cores on node
<code class="docutils literal notranslate"><span class="pre">node1</span></code>, whereas:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>prun<span class="w"> </span>--host<span class="w"> </span>node1<span class="w"> </span>--use-hwthread-cpus<span class="w"> </span>./a.out
</pre></div>
</div>
<p>launches a number of processes equal to the number of hardware
threads on <code class="docutils literal notranslate"><span class="pre">node1</span></code>.</p>
<p>When PRRTE applications are invoked in an environment managed by a
resource manager (e.g., inside of a Slurm job), and PRRTE was built
with appropriate support for that resource manager, then PRRTE will
be informed of the number of slots for each node by the resource
manager. For example:</p>
<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>shell$<span class="w"> </span>prun<span class="w"> </span>./a.out
</pre></div>
</div>
<p>launches one process for every slot (on every node) as dictated by
the resource manager job specification.</p>
<p>Also note that the one-process-per-slot restriction can be overridden
in unmanaged environments (e.g., when using hostfiles without a
resource manager) if oversubscription is enabled (by default, it is
disabled). Most parallel applications and HPC environments do not
oversubscribe; for simplicity, the majority of this documentation
assumes that oversubscription is not enabled.</p>
<div class="section" id="slots-are-not-hardware-resources">
<h2><span class="section-number">8.2.1. </span>Slots are not hardware resources<a class="headerlink" href="#slots-are-not-hardware-resources" title="Permalink to this heading"></a></h2>
<p>Slots are frequently incorrectly conflated with hardware resources.
It is important to realize that slots are an entirely different metric
than the number (and type) of hardware resources available.</p>
<p>Here are some examples that may help illustrate the difference:</p>
<ol class="arabic">
<li><p>More processor cores than slots: Consider a resource manager job
environment that tells PRRTE that there is a single node with 20
processor cores and 2 slots available. By default, PRRTE will
only let you run up to 2 processes.</p>
<p>Meaning: you run out of slots long before you run out of processor
cores.</p>
</li>
<li><p>More slots than processor cores: Consider a hostfile with a single
node listed with a <code class="docutils literal notranslate"><span class="pre">slots=50</span></code> qualification. The node has 20
processor cores. By default, PRRTE will let you run up to 50
processes.</p>
<p>Meaning: you can run many more processes than you have processor
cores.</p>
</li>
</ol>
</div>
</div>
<div class="section" id="definition-of-processor-element">
<h1><span class="section-number">8.3. </span>Definition of “processor element”<a class="headerlink" href="#definition-of-processor-element" title="Permalink to this heading"></a></h1>
<p>By default, PRRTE defines that a “processing element” is a processor
core. However, if <code class="docutils literal notranslate"><span class="pre">--use-hwthread-cpus</span></code> is specified on the command
line, then a “processing element” is a hardware thread.</p>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="index.html" class="btn btn-neutral float-left" title="8. Process placement" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="examples.html" class="btn btn-neutral float-right" title="8.4. Examples" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>

View File

@@ -0,0 +1,197 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>8.8. Rankfiles &mdash; PMIx Reference Run Time Environment 3.0.8 documentation</title>
<link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="../_static/css/theme.css" />
<!--[if lt IE 9]>
<script src="../_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
<script src="../_static/jquery.js"></script>
<script src="../_static/underscore.js"></script>
<script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="../_static/doctools.js"></script>
<script src="../_static/sphinx_highlight.js"></script>
<script src="../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../genindex.html" />
<link rel="search" title="Search" href="../search.html" />
<link rel="next" title="8.9. Deprecated options" href="deprecated.html" />
<link rel="prev" title="8.7. Diagnostics" href="diagnostics.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../index.html" class="icon icon-home">
PMIx Reference Run Time Environment
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../quickstart.html">1. Quick start</a></li>
<li class="toctree-l1"><a class="reference internal" href="../release-notes.html">2. Release Notes</a></li>
<li class="toctree-l1"><a class="reference internal" href="../getting-help.html">3. Getting help</a></li>
<li class="toctree-l1"><a class="reference internal" href="../install.html">4. Installing PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../configuration.html">5. PRRTE DVM Configuration</a></li>
<li class="toctree-l1"><a class="reference internal" href="../how-things-work/index.html">6. How Things Work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../hosts/index.html">7. Host specification</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">8. Process placement</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="overview.html">8.1. Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-slot">8.2. Definition of ‘slot’</a></li>
<li class="toctree-l2"><a class="reference internal" href="overview.html#definition-of-processor-element">8.3. Definition of “processor element”</a></li>
<li class="toctree-l2"><a class="reference internal" href="examples.html">8.4. Examples</a></li>
<li class="toctree-l2"><a class="reference internal" href="fundamentals.html">8.5. Fundamentals</a></li>
<li class="toctree-l2"><a class="reference internal" href="limits.html">8.6. Overloading and Oversubscribing</a></li>
<li class="toctree-l2"><a class="reference internal" href="diagnostics.html">8.7. Diagnostics</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">8.8. Rankfiles</a></li>
<li class="toctree-l2"><a class="reference internal" href="deprecated.html">8.9. Deprecated options</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../notifications.html">9. Notifications</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session-directory.html">10. Session directory</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developers/index.html">11. Developer’s guide</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributing.html">12. Contributing to PRRTE</a></li>
<li class="toctree-l1"><a class="reference internal" href="../license.html">13. License</a></li>
<li class="toctree-l1"><a class="reference internal" href="../man/index.html">14. PRRTE manual pages</a></li>
<li class="toctree-l1"><a class="reference internal" href="../versions.html">15. Software Version Numbers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../news/index.html">16. News</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../index.html">PMIx Reference Run Time Environment</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="index.html"><span class="section-number">8. </span>Process placement</a></li>
<li class="breadcrumb-item active"><span class="section-number">8.8. </span>Rankfiles</li>
<li class="wy-breadcrumbs-aside">
<a href="../_sources/placement/rankfiles.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<style>
.wy-table-responsive table td,.wy-table-responsive table th{white-space:normal}
</style><div class="section" id="rankfiles">
<h1><span class="section-number">8.8. </span>Rankfiles<a class="headerlink" href="#rankfiles" title="Permalink to this heading"></a></h1>
<p>Another way to specify arbitrary mappings is with a rankfile, which
gives you detailed control over process binding as well.</p>
<p>Rankfiles are text files that specify detailed information about how
individual processes should be mapped to nodes, and to which
processor(s) they should be bound. Each line of a rankfile specifies
the location of one process. The general form of each line in the
rankfile is:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rank</span> <span class="o">&lt;</span><span class="n">N</span><span class="o">&gt;=&lt;</span><span class="n">hostname</span><span class="o">&gt;</span> <span class="n">slot</span><span class="o">=&lt;</span><span class="n">slot</span> <span class="nb">list</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myrankfile
rank 0=aa slot=10-12
rank 1=bb slot=0,1,4
rank 2=cc slot=1-2
$ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
</pre></div>
</div>
<p>Means that:</p>
<ul class="simple">
<li><p>Rank 0 runs on node aa, bound to logical cores 10-12.</p></li>
<li><p>Rank 1 runs on node bb, bound to logical cores 0, 1, and 4.</p></li>
<li><p>Rank 2 runs on node cc, bound to logical cores 1 and 2.</p></li>
</ul>
<p>Similarly:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myrankfile
rank 0=aa slot=1:0-2
rank 1=bb slot=0:0,1,4
rank 2=cc slot=1-2
$ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
</pre></div>
</div>
<p>Means that:</p>
<ul class="simple">
<li><p>Rank 0 runs on node aa, bound to logical package 1, cores 10-12 (the
0th through 2nd cores on that package).</p></li>
<li><p>Rank 1 runs on node bb, bound to logical package 0, cores 0, 1,
and 4.</p></li>
<li><p>Rank 2 runs on node cc, bound to logical cores 1 and 2.</p></li>
</ul>
<p>The hostnames listed above are “absolute,” meaning that actual
resolvable hostnames are specified. However, hostnames can also be
specified as “relative,” meaning that they are specified in relation
to an externally-specified list of hostnames (e.g., by <code class="docutils literal notranslate"><span class="pre">prun</span></code>’s
<code class="docutils literal notranslate"><span class="pre">--host</span></code> argument, a hostfile, or a job scheduler).</p>
<p>The “relative” specification is of the form “<code class="docutils literal notranslate"><span class="pre">+n&lt;X&gt;</span></code>”, where <code class="docutils literal notranslate"><span class="pre">X</span></code>
is an integer specifying the Xth hostname in the set of all available
hostnames, indexed from 0. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ cat myrankfile
rank 0=+n0 slot=10-12
rank 1=+n1 slot=0,1,4
rank 2=+n2 slot=1-2
$ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
</pre></div>
</div>
<p>All package/core slot locations are be specified as <em>logical</em>
indexes. You can use tools such as HWLOC’s <code class="docutils literal notranslate"><span class="pre">lstopo</span></code> to find the
logical indexes of packages and cores.</p>
</div>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="diagnostics.html" class="btn btn-neutral float-left" title="8.7. Diagnostics" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="deprecated.html" class="btn btn-neutral float-right" title="8.9. Deprecated options" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2003-2025, The PRRTE Community.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>