114 lines
4.3 KiB
ReStructuredText
114 lines
4.3 KiB
ReStructuredText
.. -*- rst -*-
|
|
|
|
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
|
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
|
|
|
$COPYRIGHT$
|
|
|
|
Additional copyrights may follow
|
|
|
|
$HEADER$
|
|
|
|
.. The following line is included so that Sphinx won't complain
|
|
about this file not being directly included in some toctree
|
|
|
|
Overview
|
|
========
|
|
|
|
PRRTE provides a set of three controls for assigning process
|
|
locations and ranks:
|
|
|
|
#. Mapping: Assigns a default location to each process
|
|
#. Ranking: Assigns a unique integer rank value to each process
|
|
#. Binding: Constrains each process to run on specific processors
|
|
|
|
This section provides an overview of these three controls. Unless
|
|
otherwise this behavior is shared by ``prun(1)`` (working with a PRRTE
|
|
DVM), and ``prterun(1)``. More detail about PRRTE process placement is
|
|
available in the following sections (using ``--help
|
|
placement-<section>``):
|
|
|
|
* ``examples``: some examples of the interactions between mapping,
|
|
ranking, and binding options.
|
|
|
|
* ``fundamentals``: provides deeper insight into PRRTE's mapping,
|
|
ranking, and binding options.
|
|
|
|
* ``limits``: explains the difference between *overloading* and
|
|
*oversubscribing* resources.
|
|
|
|
* ``diagnostics``: describes options for obtaining various diagnostic
|
|
reports that aid the user in verifying and tuning the placement for
|
|
a specific job.
|
|
|
|
* ``rankfiles``: explains the format and use of the rankfile mapper
|
|
for specifying arbitrary process placements.
|
|
|
|
* ``deprecated``: a list of deprecated options and their new
|
|
equivalents.
|
|
|
|
* ``all``: outputs all the placement help except for the
|
|
``deprecated`` section.
|
|
|
|
|
|
Quick Summary
|
|
-------------
|
|
|
|
The two binaries that most influence process layout are ``prte(1)``
|
|
and ``prun(1)``. The ``prte(1)`` process discovers the allocation,
|
|
establishes a Distributed Virtual Machine by starting a ``prted(1)``
|
|
daemon on each node of the allocation, and defines the efault
|
|
mapping/ranking/binding policies for all jobs. The ``prun(1)`` process
|
|
defines the specific mapping/ranking/binding for a specific job. Most
|
|
of the command line controls are targeted to ``prun(1)`` since each job
|
|
has its own unique requirements.
|
|
|
|
``prterun(1)`` is just a wrapper around ``prte(1)`` for a single job
|
|
PRRTE DVM. It is doing the job of both ``prte(1)`` and ``prun(1)``,
|
|
and, as such, accepts the sum all of their command line arguments. Any
|
|
example that uses ``prun(1)`` can substitute the use of ``prterun(1)``
|
|
except where otherwise noted.
|
|
|
|
The ``prte(1)`` process attempts to automatically discover the nodes
|
|
in the allocation by querying supported resource managers. If a
|
|
supported resource manager is not present then ``prte(1)`` relies on a
|
|
hostfile provided by the user. In the absence of such a hostfile it
|
|
will run all processes on the localhost.
|
|
|
|
If running under a supported resource manager, the ``prte(1)`` process
|
|
will start the daemon processes (``prted(1)``) on the remote nodes
|
|
using the corresponding resource manager process starter. If no such
|
|
starter is available then ``ssh`` (or ``rsh``) is used.
|
|
|
|
Minus user direction, PRRTE will automatically map processes in a
|
|
round-robin fashion by CPU, binding each process to its own CPU. The
|
|
type of CPU used (core vs hwthread) is determined by (in priority
|
|
order):
|
|
|
|
* user directive on the command line via the HWTCPUS qualifier to
|
|
the ``--map-by`` directive
|
|
|
|
* setting the ``rmaps_default_mapping_policy`` MCA parameter to
|
|
include the ``HWTCPUS`` qualifier. This parameter sets the default
|
|
value for a PRRTE DVM |mdash| qualifiers are carried across to DVM
|
|
jobs started via ``prun`` unless overridden by the user's command
|
|
line
|
|
|
|
* defaulting to ``CORE`` in topologies where core CPUs are defined,
|
|
and to ``hwthreads`` otherwise.
|
|
|
|
By default, the ranks are assigned in accordance with the mapping
|
|
directive |mdash| e.g., jobs that are mapped by-node will have the
|
|
process ranks assigned round-robin on a per-node basis.
|
|
|
|
PRRTE automatically binds processes unless directed not to do so by
|
|
the user. Minus direction, PRRTE will bind individual processes to
|
|
their own CPU within the object to which they were mapped. Should a
|
|
node become oversubscribed during the mapping process, and if
|
|
oversubscription is allowed, all subsequent processes assigned to that
|
|
node will *not* be bound.
|
|
|
|
.. include:: /prrte-rst-content/definitions-slots.rst
|
|
|
|
.. include:: /prrte-rst-content/definitions-pes.rst
|