Upgrade macos
This commit is contained in:
Bassem Girgis
@@ -0,0 +1,34 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Allow execution as root **(STRONGLY DISCOURAGED)**.
|
||||
|
||||
Running as root exposes the user to potentially catastrophic file
|
||||
system corruption and damage |mdash| e.g., if the user accidentally
|
||||
points the root of the session directory to a system required point,
|
||||
this directory and all underlying elements will be deleted upon job
|
||||
completion, thereby rendering the system inoperable.
|
||||
|
||||
It is recognized that some environments (e.g., containers) may require
|
||||
operation as root, and that the user accepts the risks in those
|
||||
scenarios. Accordingly, one can override PRRTE's run-as-root
|
||||
protection by providing one of the following:
|
||||
|
||||
* The ``--allow-run-as-root`` command line directive
|
||||
* Adding **BOTH** of the following environmental parameters:
|
||||
|
||||
* ``PRTE_ALLOW_RUN_AS_ROOT=1``
|
||||
* ``PRTE_ALLOW_RUN_AS_ROOT_CONFIRM=1``
|
||||
|
||||
Again, we recommend this only be done if absolutely necessary.
|
||||
@@ -0,0 +1,83 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
By default, processes are bound to individual CPUs (either COREs or
|
||||
HWTHREADs, as defined by default or by user specification for the
|
||||
job). On nodes that are OVERSUBSCRIBEd (i.e., where the number of
|
||||
procs exceeds the number of assigned slots), the default is to not
|
||||
bind the processes.
|
||||
|
||||
.. note:: Processes from prior jobs that are already executing on a
|
||||
node are not "unbound" when a new job mapping results in the
|
||||
node becoming oversubscribed.
|
||||
|
||||
Binding is performed to the first available specified object type
|
||||
within the object where the process was mapped. In other words,
|
||||
binding can only be done to the mapped object or to a resource
|
||||
located beneath that object.
|
||||
|
||||
An object is considered completely consumed when the number of
|
||||
processes bound to it equals the number of CPUs within it. Unbound
|
||||
processes are not considered in this computation. Additional
|
||||
processes cannot be mapped to consumed objects unless the
|
||||
``OVERLOAD`` qualifier is provided via the ``--bind-to`` command
|
||||
line option.
|
||||
|
||||
Note that directives and qualifiers are case-insensitive
|
||||
and can be shortened to the minimum number of characters
|
||||
to uniquely identify them. Thus, ``L1CACHE`` can be given
|
||||
as ``l1cache`` or simply as ``L1``.
|
||||
|
||||
Supported binding directives include:
|
||||
|
||||
* ``NONE`` does not bind the processes
|
||||
|
||||
* ``HWTHREAD`` binds each process to a single hardware
|
||||
thread/ This requires that hwthreads be treated
|
||||
as independent CPUs (i.e., that either the ``HWTCPUS``
|
||||
qualifier be provided to the ``map-by`` option or
|
||||
that ``hwthreads`` be designated as CPUs by default).
|
||||
|
||||
* ``CORE`` binds each process to a single core. This
|
||||
can be done whether ``hwthreads`` or ``cores`` are being
|
||||
treated as independent CPUs provided that mapping
|
||||
is performed at the core or higher level.
|
||||
|
||||
* ``L1CACHE`` binds each process to all the CPUs in
|
||||
an ``L1`` cache.
|
||||
|
||||
* ``L2CACHE`` binds each process to all the CPUs in
|
||||
an ``L2`` cache
|
||||
|
||||
* ``L3CACHE`` binds each process to all the CPUs in
|
||||
an ``L3`` cache
|
||||
|
||||
* ``NUMA`` binds each process to all the CPUs in a ``NUMA``
|
||||
region
|
||||
|
||||
* ``PACKAGE`` binds each process to all the CPUs in a ``PACKAGE``
|
||||
|
||||
Any directive can include qualifiers by adding a colon (:) and any
|
||||
combination of one or more of the following to the ``--bind-to``
|
||||
option:
|
||||
|
||||
* ``OVERLOAD`` indicates that objects can have more
|
||||
processes bound to them than CPUs within them
|
||||
|
||||
* ``IF-SUPPORTED`` indicates that the job should continue to
|
||||
be launched and executed even if binding cannot be
|
||||
performed as requested.
|
||||
|
||||
.. note:: Directives and qualifiers are case-insensitive.
|
||||
``OVERLOAD`` is the same as ``overload``.
|
||||
@@ -0,0 +1,30 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Host syntax consists of a comma-delimited list of node names, each
|
||||
entry optionally containing a ``:N`` extension indicating the number
|
||||
of slots to assign to that entry:
|
||||
|
||||
.. code::
|
||||
|
||||
--host node01:5,node02
|
||||
|
||||
In the absence of the slot extension, one slot will be assigned to the
|
||||
node. Duplicate entries are aggregated and the number of slots
|
||||
assigned to that node are summed together.
|
||||
|
||||
.. note:: A "slot" is the PRRTE term for an allocatable unit where we
|
||||
can launch a process. Thus, the number of slots equates to the
|
||||
maximum number of processes PRRTE may start on that node without
|
||||
oversubscribing it.
|
||||
@@ -0,0 +1,23 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Debug daemon output is enabled and all output from the daemons is
|
||||
redirected into files with names of the form:
|
||||
|
||||
.. code::
|
||||
|
||||
output-prted-<daemon-nspace>-<nodename>.log
|
||||
|
||||
These names avoid conflict on shared file systems. The files are
|
||||
located in the top-level session directory assigned to the DVM.
|
||||
@@ -0,0 +1,17 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Debug daemon output enabled. This is a somewhat limited stream of
|
||||
information normally used to simply confirm that the daemons
|
||||
started. Includes leaving the output streams open.
|
||||
@@ -0,0 +1,53 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
The ``display`` command line directive must be accompanied by a
|
||||
comma-delimited list of case-insensitive options indicating what
|
||||
information about the job and/or allocation is to be displayed. The
|
||||
full directive need not be provided |mdash| only enough characters are
|
||||
required to uniquely identify the directive. For example, ``ALL`` is
|
||||
sufficient to represent the ``ALLOCATION`` directive |mdash| while ``MAP``
|
||||
can not be used to represent ``MAP-DEVEL`` (though ``MAP-D`` would
|
||||
suffice).
|
||||
|
||||
Supported values include:
|
||||
|
||||
* ``ALLOCATION`` displays the detected hosts and slot assignments for
|
||||
this job
|
||||
|
||||
* ``BINDINGS`` displays the resulting bindings applied to processes in
|
||||
this job
|
||||
|
||||
* ``MAP`` displays the resulting locations assigned to processes in
|
||||
this job
|
||||
|
||||
* ``MAP-DEVEL`` displays a more detailed report on the locations
|
||||
assigned to processes in this job that includes local and node
|
||||
ranks, assigned bindings, and other data
|
||||
|
||||
* ``TOPO=LIST`` displays the topology of each node in the
|
||||
semicolon-delimited list that is allocated to the job
|
||||
|
||||
* ``CPUS[=LIST]`` displays the available CPUs on the provided
|
||||
semicolon-delimited list of nodes (defaults to all nodes)
|
||||
|
||||
The display command line directive can include qualifiers by adding a
|
||||
colon (``:``) and any combination of one or more of the following
|
||||
(delimited by colons):
|
||||
|
||||
* ``PARSEABLE`` directs that the output be provided in a format that
|
||||
is easily parsed by machines. Note that ``PARSABLE`` is also accepted as
|
||||
a typical spelling for the qualifier.
|
||||
|
||||
Provided qualifiers will apply to *all* of the display directives.
|
||||
@@ -0,0 +1,90 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
PRRTE supports several levels of user-specified host lists based on an
|
||||
established precedence order. Users can specify a default hostfile
|
||||
that contains a list of nodes to be used by the DVM. Only one default
|
||||
hostfile can be provided for a given DVM. In addition, users can
|
||||
specify a hostfile that contains a list of nodes to be used for a DVM,
|
||||
or can provide a comma-delimited list of nodes to be used for that DVM
|
||||
via the ``--host`` command line option.
|
||||
|
||||
The precedence order applied to these various options depends to some
|
||||
extent on the local environment. The following table illustrates how
|
||||
host and hostfile directives work together to define the set of hosts
|
||||
upon which a DVM will execute in the absence of a resource manager
|
||||
(RM):
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 10 7 10 32
|
||||
|
||||
* - Default hostfile
|
||||
- host
|
||||
- hostfile
|
||||
- Result
|
||||
|
||||
* - unset
|
||||
- unset
|
||||
- unset
|
||||
- | The DVN will consist solely of the
|
||||
| local host where the DVM
|
||||
| was started.
|
||||
|
||||
* - unset
|
||||
- set
|
||||
- unset
|
||||
- | Host option defines resource list for the DVM.
|
||||
|
||||
* - unset
|
||||
- unset
|
||||
- set
|
||||
- | Hostfile option defines resource list for the DVM.
|
||||
|
||||
* - unset
|
||||
- set
|
||||
- set
|
||||
- | Hostfile option defines resource list for the DVM,
|
||||
| then host filters the list to define the final
|
||||
| set of nodes to be used by the DVM
|
||||
|
||||
* - set
|
||||
- unset
|
||||
- unset
|
||||
- | Default hostfile defines resource list for the DVM
|
||||
|
||||
* - set
|
||||
- set
|
||||
- unset
|
||||
- | Default hostfile defines resource list for the DVM,
|
||||
| then host filters the list to define the final
|
||||
| set of nodes to be used by the DVM
|
||||
|
||||
* - set
|
||||
- set
|
||||
- set
|
||||
- | Default hostfile defines resource list for the DVM,
|
||||
| then hostfile filters the list, and then host filters
|
||||
| the list to define the final set of nodes to be
|
||||
| used by the DVM
|
||||
|
||||
This changes somewhat in the presence of an RM as that entity
|
||||
specifies the initial allocation of nodes. In this case, the default
|
||||
hostfile, hostfile and host directives are all used to filter the RM's
|
||||
specification so that a user can utilize different portions of the
|
||||
allocation for different DVMs. This is done according to the same
|
||||
precedence order as in the prior table, with the RM providing the
|
||||
initial pool of nodes.
|
||||
|
||||
.. include:: /prrte-rst-content/detail-hostfiles.rst
|
||||
@@ -0,0 +1,59 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
A required argument is passed to the ``--dvm`` directive to specify
|
||||
the location of the DVM controller (e.g., ``--dvm pid:12345``) or by
|
||||
passing the string ``search`` to instead search for an existing
|
||||
controller.
|
||||
|
||||
Supported options include:
|
||||
|
||||
* ``search``: directs the tool to search for available DVM controllers
|
||||
it is authorized to use, connecting to the first such candidate it
|
||||
finds.
|
||||
|
||||
* ``pid:<arg>``: provides the PID of the target DVM controller. This
|
||||
can be given as either the PID itself (arg = int) or the path to a
|
||||
file that contains the PID (arg = ``file:<path>``)
|
||||
|
||||
* ``file:<path>``: provides the path to a PMIx rendezvous file that is
|
||||
output by PMIx servers |mdash| the file contains all the required
|
||||
information for completing the connection
|
||||
|
||||
* ``uri:<arg>``: specifies the URI of the DVM controller, or the name of
|
||||
the file (specified as ``file:filename``) that contains that info
|
||||
|
||||
* ``ns:<arg>``: specifies the namespace of the DVM controller
|
||||
|
||||
* ``system``: exclusively find and use the system-level DVM controller
|
||||
|
||||
* ``system-first``: look for a system-level DVM controller, fall back
|
||||
to searching for an available DVM controller the command is
|
||||
authorized to use if a system-level controller is not found
|
||||
|
||||
Examples:
|
||||
|
||||
.. code::
|
||||
|
||||
prterun --dvm file:dvm_uri.txt --np 4 ./a.out
|
||||
|
||||
prterun --dvm pid:12345 --np 4 ./a.out
|
||||
|
||||
prterun --dvm uri:file:dvm_uri.txt --np 4 ./a.out
|
||||
|
||||
prterun --dvm ns:prte-node1-2095 --np 4 ./a.out
|
||||
|
||||
prterun --dvm pid:file:prte_pid.txt --np 4 ./a.out
|
||||
|
||||
prterun --dvm search --np 4 ./a.out
|
||||
@@ -0,0 +1,18 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Comma-delimited list of additional signals (names or integers) to
|
||||
forward to application processes (``none`` = forward
|
||||
nothing). Signals provided by default include SIGTSTP, SIGUSR1,
|
||||
SIGUSR2, SIGABRT, SIGALRM, and SIGCONT.
|
||||
@@ -0,0 +1,56 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
PRRTE supports several levels of user-specified hostfiles based on an
|
||||
established precedence order. Users can specify a hostfile that
|
||||
contains a list of nodes to be used for the job, or can provide a
|
||||
comma-delimited list of nodes to be used for that job via the
|
||||
``--host`` command line option.
|
||||
|
||||
The precedence order applied to these various options depends to some
|
||||
extent on the local environment. The following table illustrates how
|
||||
host and hostfile directives work together to define the set of hosts
|
||||
upon which a DVM will execute the job in the absence of a resource
|
||||
manager (RM):
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 7 10 45
|
||||
|
||||
* - host
|
||||
- hostfile
|
||||
- Result
|
||||
|
||||
* - unset
|
||||
- unset
|
||||
- | The DVM will utilize all its available resources
|
||||
| when mapping the job.
|
||||
|
||||
* - set
|
||||
- unset
|
||||
- | Host option defines resource list for the job
|
||||
|
||||
* - unset
|
||||
- set
|
||||
- | Hostfile defines resource list for the job
|
||||
|
||||
* - set
|
||||
- set
|
||||
- | Hostfile defines resource list for the job,
|
||||
| then host filters the list to define the final
|
||||
| set of nodes to be used for the job
|
||||
|
||||
.. include:: /prrte-rst-content/detail-hostfiles.rst
|
||||
|
||||
.. include:: /prrte-rst-content/detail-hosts-relative-indexing.rst
|
||||
@@ -0,0 +1,19 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Do not discard stdout/stderr of remote PRRTE daemons. The primary use
|
||||
for this option is to ensure that the daemon output streams (i.e.,
|
||||
stdout and stderr) remain open after launch, thus allowing the user to
|
||||
see any daemon-generated error messages. Otherwise, the daemon will
|
||||
"daemonize" itself upon launch, thereby closing its output streams.
|
||||
@@ -0,0 +1,139 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Processes are mapped based on one of the following directives as
|
||||
applied at the job level:
|
||||
|
||||
* ``SLOT`` assigns procs to each node up to the number of available
|
||||
slots on that node before moving to the next node in the
|
||||
allocation
|
||||
|
||||
* ``HWTHREAD`` assigns a proc to each hardware thread on a node in a
|
||||
round-robin manner up to the number of available slots on that
|
||||
node before moving to the next node in the allocation
|
||||
|
||||
* ``CORE`` (default) assigns a proc to each core on a node in a
|
||||
round-robin manner up to the number of available slots on that
|
||||
node before moving to the next node in the allocation
|
||||
|
||||
* ``L1CACHE`` assigns a proc to each L1 cache on a node in a
|
||||
round-robin manner up to the number of available slots on that
|
||||
node before moving to the next node in the allocation
|
||||
|
||||
* ``L2CACHE`` assigns a proc to each L2 cache on a node in a
|
||||
round-robin manner up to the number of available slots on that
|
||||
node before moving to the next node in the allocation
|
||||
|
||||
* ``L3CACHE`` assigns a proc to each L3 cache on a node in a
|
||||
round-robin manner up to the number of available slots on that
|
||||
node before moving to the next node in the allocation
|
||||
|
||||
* ``NUMA`` assigns a proc to each NUMA region on a node in a
|
||||
round-robin manner up to the number of available slots on that
|
||||
node before moving to the next node in the allocation
|
||||
|
||||
* ``PACKAGE`` assigns a proc to each package on a node in a
|
||||
round-robin manner up to the number of available slots on that
|
||||
node before moving to the next node in the allocation
|
||||
|
||||
* ``NODE`` assigns processes in a round-robin fashion to all nodes
|
||||
in the allocation, with the number assigned to each node capped
|
||||
by the number of available slots on that node
|
||||
|
||||
* ``SEQ`` (often accompanied by the file=<path> qualifier) assigns
|
||||
one process to each node specified in the file. The sequential
|
||||
file is to contain an entry for each desired process, one per
|
||||
line of the file.
|
||||
|
||||
* ``PPR:N``:resource maps N procs to each instance of the specified
|
||||
resource type in the allocation
|
||||
|
||||
* ``RANKFILE`` (often accompanied by the file=<path> qualifier) assigns
|
||||
one process to the node/resource specified in each entry of the
|
||||
file, one per line of the file.
|
||||
|
||||
* ``PE-LIST=a,b`` assigns procs to each node in the allocation based on
|
||||
the ORDERED qualifier. The list is comprised of comma-delimited
|
||||
ranges of CPUs to use for this job. If the ORDERED qualifier is not
|
||||
provided, then each node will be assigned procs up to the number of
|
||||
available slots, capped by the availability of the specified CPUs.
|
||||
If ORDERED is given, then one proc will be assigned to each of the
|
||||
specified CPUs, if available, capped by the number of slots on each
|
||||
node and the total number of specified processes. Providing the
|
||||
OVERLOAD qualifier to the "bind-to" option removes the check on
|
||||
availability of the CPU in both cases.
|
||||
|
||||
Any directive can include qualifiers by adding a colon (``:``) and any
|
||||
combination of one or more of the following (delimited by colons) to
|
||||
the ``--map-by`` option (except where noted):
|
||||
|
||||
* ``PE=n`` bind n CPUs to each process (can not be used in combination
|
||||
with rankfile or pe-list directives)
|
||||
|
||||
* ``SPAN`` load balance the processes across the allocation by treating
|
||||
the allocation as a single "super-node" (can not be used in
|
||||
combination with ``slot``, ``node``, ``seq``, ``ppr``, ``rankfile``, or
|
||||
``pe-list`` directives)
|
||||
|
||||
* ``OVERSUBSCRIBE`` allow more processes on a node than processing elements
|
||||
|
||||
* ``NOOVERSUBSCRIBE`` means ``!OVERSUBSCRIBE``
|
||||
|
||||
* ``NOLOCAL`` do not launch processes on the same node as ``prun``
|
||||
|
||||
* ``HWTCPUS`` use hardware threads as CPU slots
|
||||
|
||||
* ``CORECPUS`` use cores as CPU slots (default)
|
||||
|
||||
* ``INHERIT`` indicates that a child job (i.e., one spawned from within
|
||||
an application) shall inherit the placement policies of the parent job
|
||||
that spawned it.
|
||||
|
||||
* ``NOINHERIT`` means ```!INHERIT``
|
||||
|
||||
* ``FILE=<path>`` (path to file containing sequential or rankfile entries).
|
||||
|
||||
* ``ORDERED`` only applies to the ``PE-LIST`` option to indicate that
|
||||
procs are to be bound to each of the specified CPUs in the order in
|
||||
which they are assigned (i.e., the first proc on a node shall be
|
||||
bound to the first CPU in the list, the second proc shall be bound
|
||||
to the second CPU, etc.)
|
||||
|
||||
.. note:: Directives and qualifiers are case-insensitive and can be
|
||||
shortened to the minimum number of characters to uniquely
|
||||
identify them. Thus, ``L1CACHE`` can be given as ``l1cache`` or
|
||||
simply as ``L1``.
|
||||
|
||||
The type of CPU (core vs hwthread) used in the mapping algorithm
|
||||
is determined as follows:
|
||||
|
||||
* by user directive on the command line via the HWTCPUS qualifier to
|
||||
the ``--map-by`` directive
|
||||
|
||||
* by setting the ``rmaps_default_mapping_policy`` MCA parameter to
|
||||
include the ``HWTCPUS`` qualifier. This parameter sets the default
|
||||
value for a PRRTE DVM |mdash| qualifiers are carried across to DVM jobs
|
||||
started via ``prun`` unless overridden by the user's command line
|
||||
|
||||
* defaults to CORE in topologies where core CPUs are defined, and to
|
||||
hwthreads otherwise.
|
||||
|
||||
If your application uses threads, then you probably want to ensure that
|
||||
you are either not bound at all (by specifying ``--bind-to none``), or
|
||||
bound to multiple cores using an appropriate binding level or specific
|
||||
number of processing elements per application process via the ``PE=#``
|
||||
qualifier to the ``--map-by`` command line directive.
|
||||
|
||||
A more detailed description of the mapping, ranking, and binding
|
||||
procedure can be obtained via the ``--help placement`` option.
|
||||
@@ -0,0 +1,19 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Disable automatic ``--prefix`` behavior. PRRTE automatically sets the
|
||||
prefix for remote daemons if it was either configured with the
|
||||
``--enable-prte-prefix-by-default`` option OR prte itself was executed
|
||||
with an absolute path to the ``prte`` command. This option disables
|
||||
that behavior.
|
||||
@@ -0,0 +1,55 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
The ``output`` command line directive must be accompanied by a
|
||||
comma-delimited list of case-insensitive options that control how
|
||||
output is generated. The full directive need not be provided |mdash| only
|
||||
enough characters are required to uniquely identify the directive. For
|
||||
example, ``MERGE`` is sufficient to represent the
|
||||
``MERGE-STDERR-TO-STDOUT`` directive |mdash| while ``TAG`` can not be
|
||||
used to represent ``TAG-DETAILED`` (though ``TAG-D`` would suffice).
|
||||
|
||||
Supported values include:
|
||||
|
||||
* ``TAG`` marks each output line with the ``[job,rank]<stream>:`` of
|
||||
the process that generated it
|
||||
|
||||
* ``TAG-DETAILED`` marks each output line with a detailed annotation
|
||||
containing ``[namespace,rank][hostname:pid]<stream>:`` of the
|
||||
process that generated it
|
||||
|
||||
* ``TAG-FULLNAME`` marks each output line with the
|
||||
``[namespace,rank]<stream>:`` of the process that generated it
|
||||
|
||||
* ``TAG-FULLNAME`` marks each output line with the
|
||||
``[namespace,rank]<stream>:`` of the process that generated it
|
||||
|
||||
* ``TIMESTAMP`` prefixes each output line with a ``[datetime]<stream>:``
|
||||
stamp. Note that the timestamp will be the time when the line is
|
||||
output by the DVM and not the time when the source output it
|
||||
|
||||
* ``XML`` provides all output in a pseudo-XML format
|
||||
``MERGE-STDERR-TO-STDOUT`` merges stderr into stdout
|
||||
|
||||
* ``DIR=DIRNAME`` redirects output from application processes into
|
||||
``DIRNAME/job/rank/std[out,err,diag]``. The provided name will be
|
||||
converted to an absolute path
|
||||
|
||||
* ``FILE=FILENAME`` redirects output from application processes into
|
||||
``filename.rank.`` The provided name will be converted to an absolute
|
||||
path
|
||||
|
||||
Supported qualifiers include ``NOCOPY`` (do not copy the output to the
|
||||
stdout/err streams), and ``RAW`` (do not buffer the output into complete
|
||||
lines, but instead output it as it is received).
|
||||
@@ -0,0 +1,21 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Specify the personality to be used. This governs selection of the
|
||||
plugin responsible for defining and parsing the command line,
|
||||
harvesting and forwarding environmental variables, and providing
|
||||
library-dependent support to the launched processes. Examples include
|
||||
``ompi`` for an application compiled with Open MPI, ``mpich`` for one
|
||||
built against the MPICH library, or ``oshmem`` for an OpenSHMEM
|
||||
application compiled against SUNY's reference library.
|
||||
@@ -0,0 +1,18 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Pass a PMIx MCA parameter
|
||||
|
||||
Syntax: ``--pmixmca <key> <value>``, where ``key`` is the parameter
|
||||
name and ``value`` is the parameter value.
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Prefix to be used to look for PRRTE executables. PRRTE automatically
|
||||
sets the prefix for remote daemons if it was either configured with
|
||||
the ``--enable-prte-prefix-by-default`` option OR prte itself was
|
||||
executed with an absolute path to the prte command. This option
|
||||
overrides those settings, if present, and forces use of the provided
|
||||
path.
|
||||
@@ -0,0 +1,18 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Pass a PRRTE MCA parameter.
|
||||
|
||||
Syntax: ``--prtemca <key> <value>``, where ``key`` is the parameter
|
||||
name and ``value`` is the parameter value.
|
||||
@@ -0,0 +1,66 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
PRRTE automatically ranks processes for each job starting from zero.
|
||||
Regardless of the algorithm used, rank assignments span applications
|
||||
in the same job |mdash| i.e., a command line of
|
||||
|
||||
.. code::
|
||||
|
||||
-n 3 app1 : -n 2 app2
|
||||
|
||||
will result in ``app1`` having three processes ranked 0-2 and ``app2``
|
||||
having two processes ranked 3-4.
|
||||
|
||||
By default, process ranks are assigned in accordance with the mapping
|
||||
directive |mdash| e.g., jobs that are mapped by-node will have the process
|
||||
ranks assigned round-robin on a per-node basis. However, users can override
|
||||
the default by specifying any of the following directives using the
|
||||
``--rank-by`` command line option:
|
||||
|
||||
* ``SLOT`` assigns ranks to each process on a node in the order in
|
||||
which the mapper assigned them. This is the default behavior,
|
||||
but is provided as an explicit option to allow users to override
|
||||
any alternative default specified in the environment. When mapping
|
||||
to a specific resource type, procs assigned to a given instance
|
||||
of that resource on a node will be ranked on a per-resource basis
|
||||
on that node before moving to the next node.
|
||||
|
||||
* ``NODE`` assigns ranks round-robin on a per-node basis
|
||||
|
||||
* ``FILL`` assigns ranks to procs mapped to a particular resource type
|
||||
on each node, filling all ranks on that resource before moving to
|
||||
the next resource on that node. For example, procs mapped by
|
||||
``L1cache`` would have all procs on the first ``L1cache`` ranked
|
||||
sequentially before moving to the second ``L1cache`` on the
|
||||
node. Once all procs on the node have been ranked, ranking would
|
||||
continue on the next node.
|
||||
|
||||
* ``SPAN`` assigns ranks round-robin to procs mapped to a particular
|
||||
resource type, treating the collection of resource instances
|
||||
spanning the entire allocation as a single "super node" before
|
||||
looping around for the next pass. Thus, ranking would begin with the
|
||||
first proc on the first ``L1cache`` on the first node, then the next
|
||||
rank would be assigned to the first proc on the second ``L1cache``
|
||||
on that node, proceeding across until the first proc had been ranked
|
||||
on all ``L1cache`` used by the job before circling around to rank
|
||||
the second proc on each object.
|
||||
|
||||
The ``rank-by`` command line option has no qualifiers.
|
||||
|
||||
.. note:: Directives are case-insensitive. ``SPAN`` is the same as
|
||||
``span``.
|
||||
|
||||
A more detailed description of the mapping, ranking, and binding
|
||||
procedure can be obtained via the ``--help placement`` option.
|
||||
@@ -0,0 +1,167 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
The ``--runtime-options`` command line directive must be accompanied
|
||||
by a comma-delimited list of case-insensitive options that control the
|
||||
runtime behavior of the job. The full directive need not be provided
|
||||
|mdash| only enough characters are required to uniquely identify the
|
||||
directive.
|
||||
|
||||
Runtime options are typically ``true`` or ``false``, though this is
|
||||
not a requirement on developers. Since the value of each option may
|
||||
need to be set (e.g., to override a default set by MCA parameter), the
|
||||
syntax of the command line directive includes the use of an ``=``
|
||||
character to allow inclusion of a value for the option. For example,
|
||||
one can set the ``ABORT-NONZERO-STATUS`` option to ``true`` by
|
||||
specifying it as ``ABORT-NONZERO-STATUS=1``. Note that boolean options
|
||||
can be set to ``true`` using a non-zero integer or a case-insensitive
|
||||
string of the word ``true``. For the latter representation, the user
|
||||
need only provide at least the ``T`` character. The same policy
|
||||
applies to setting a boolean option to ``false``.
|
||||
|
||||
Note that a boolean option will default to ``true`` if provided
|
||||
without a value. Thus, ``--runtime-options abort-nonzero`` is
|
||||
sufficient to set the ``ABORT-NONZERO-STATUS`` option to ``true``.
|
||||
|
||||
Supported values include:
|
||||
|
||||
* ``ERROR-NONZERO-STATUS[=(bool)]``: if set to false, this directs the
|
||||
runtime to treat a process that exits with non-zero status as a
|
||||
normal termination. If set to true, the runtime will consider such
|
||||
an occurrence as an error termination and take appropriate action
|
||||
|mdash| i.e., the job will be terminated unless a runtime option
|
||||
directs otherwise. This option defaults to a true value if the
|
||||
option is given without a value.
|
||||
|
||||
* ``DONOTLAUNCH``: directs the runtime to map but not launch the
|
||||
specified job. This is provided to help explore possible process
|
||||
placement patterns before actually starting execution. No value need
|
||||
be passed as this is not an option that can be set by default in
|
||||
PRRTE.
|
||||
|
||||
* ``SHOW-PROGRESS[=(bool)]``: requests that the runtime provide
|
||||
progress reports on its startup procedure |mdash| i.e., the launch
|
||||
of its daemons in support of a job. This is typically used to debug
|
||||
DVM startup on large systems. This option defaults to a true value
|
||||
if the option is given without a value.
|
||||
|
||||
* ``NOTIFYERRORS[=(bool)]``: if set to true, requests that the runtime
|
||||
provide a PMIx event whenever a job encounters an error |mdash|
|
||||
e.g., a process fails. The event is to be delivered to each
|
||||
remaining process in the job. This option defaults to a true value
|
||||
if the option is given without a value. See ``--help
|
||||
notifications`` for more detail as to the PMIx event codes available
|
||||
for capturing failure events.
|
||||
|
||||
* ``RECOVERABLE[=(bool)]``: if set to true, this indicates that the
|
||||
application wishes to consider the job as recoverable |mdash| i.e.,
|
||||
the application is assuming responsibility for recovering from any
|
||||
process failure. This could include application-driven spawn of a
|
||||
substitute process or internal compensation for the missing
|
||||
process. This option defaults to a true value if the option is given
|
||||
without a value.
|
||||
|
||||
* ``AUTORESTART[=(bool)]``: if set to true, this requests that the
|
||||
runtime automatically restart failed processes up to "max restarts"
|
||||
number of times. This option defaults to a true value if the option
|
||||
is given without a value.
|
||||
|
||||
* ``CONTINUOUS[=(bool)]``: if set to true, this informs the runtime
|
||||
that the processes in this job are to run until explicitly
|
||||
terminated. Processes that fail are to be automatically restarted up
|
||||
to "max restarts" number of times. Notification of process failure
|
||||
is to be delivered to all processes in the application. This is the
|
||||
equivalent of specifying ``RECOVERABLE``, ``NOTIFYERRORS``, and
|
||||
``AUTORESTART`` options except that the runtime, not the
|
||||
application, assumes responsibility for process recovery. This
|
||||
option defaults to a true value if the option is given without a
|
||||
value.
|
||||
|
||||
* ``MAX-RESTARTS=<int>``: indicates the maximum number of times a
|
||||
given process is to be restarted. This can be set at the application
|
||||
or job level (which will then apply to all applications in that
|
||||
job).
|
||||
|
||||
* ``EXEC-AGENT=<path>`` indicates the executable that shall be used to
|
||||
start an application process. The resulting command for starting an
|
||||
application process will be ``<path> app <app-argv>``. The path may
|
||||
contain its own command line arguments.
|
||||
|
||||
* ``DEFAULT-EXEC-AGENT``: directs the runtime to use the system
|
||||
default exec agent to start an application process. No value need be
|
||||
passed as this is not an option that can be set by default in PRRTE.
|
||||
|
||||
* ``OUTPUT-PROCTABLE[(=channel)]``: directs the runtime to report the
|
||||
convential debugger process table (includes PID and host location of
|
||||
each process in the application). Output is directed to stdout if
|
||||
the channel is ``-``, stderr if ``+``, or into the specified file
|
||||
otherwise. If no channel is specified, output will be directed to
|
||||
stdout.
|
||||
|
||||
* ``STOP-ON-EXEC``: directs the runtime to stop the application
|
||||
process(es) immediately upon exec'ing them. The directive will apply
|
||||
to all processes in the job.
|
||||
|
||||
* ``STOP-IN-INIT``: indicates that the runtime should direct the
|
||||
application process(es) to stop in ``PMIx_Init()``. The directive
|
||||
will apply to all processes in the job.
|
||||
|
||||
* ``STOP-IN-APP``: indicates that the runtime should direct
|
||||
application processes to stop at some application-defined place and
|
||||
notify they are ready-to-debug. The directive will apply to all
|
||||
processes in the job.
|
||||
|
||||
* ``TIMEOUT=<string>``: directs the runtime to terminate the job after
|
||||
it has executed for the specified time. Time is specified in
|
||||
colon-delimited format |mdash| e.g., ``01:20:13:05`` to indicate 1
|
||||
day, 20 hours, 13 minutes and 5 seconds. Time specified without
|
||||
colons will be assumed to have been given in seconds.
|
||||
|
||||
* ``SPAWN-TIMEOUT=<string>``: directs the runtime to terminate the job
|
||||
if job launch is not completed within the specified time. Time is
|
||||
specified in colon-delimited format |mdash| e.g., ``01:20:13:05`` to
|
||||
indicate 1 day, 20 hours, 13 minutes and 5 seconds. Time specified
|
||||
without colons will be assumed to have been given in seconds.
|
||||
|
||||
* ``REPORT-STATE-ON-TIMEOUT[(=bool)]``: directs the runtime to provide
|
||||
a detailed report on job and application process state upon job
|
||||
timeout. This option defaults to a true value if the option is given
|
||||
without a value.
|
||||
|
||||
* ``GET-STACK-TRACES[(=bool)]``: requests that the runtime provide
|
||||
stack traces on all application processes still executing upon
|
||||
timeout. This option defaults to a true value if the option is given
|
||||
without a value.
|
||||
|
||||
* ``REPORT-CHILD-JOBS-SEPARATELY[(=bool)]``: directs the runtime to
|
||||
report the exit status of any child jobs spawned by the primary job
|
||||
separately. If false, then the final exit status reported will be
|
||||
zero if the primary job and all spawned jobs exit normally, or the
|
||||
first non-zero status returned by either primary or child jobs.
|
||||
This option defaults to a true value if the option is given without
|
||||
a value.
|
||||
|
||||
* ``AGGREGATE-HELP-MESSAGES[(=bool)]``: directs the runtime to
|
||||
aggregate help messages, reporting each unique help message once
|
||||
accompanied by the number of processes that reported it. This option
|
||||
defaults to a true value if the option is given without a value.
|
||||
|
||||
* ``FWD-ENVIRONMENT[(=bool)]``: directs the runtime to forward the
|
||||
entire local environment in support of the application. This option
|
||||
defaults to a true value if the option is given without a value.
|
||||
|
||||
The ``--runtime-options`` command line option has no qualifiers.
|
||||
|
||||
.. note:: Directives are case-insensitive. ``FWD-ENVIRONMENT`` is the
|
||||
same as ``fwd-environment``.
|
||||
@@ -0,0 +1,19 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Adjust buffering for stdout/stderr. Allowable values:
|
||||
|
||||
* 0: unbuffered
|
||||
* 1: line buffered
|
||||
* 2: fully buffered
|
||||
@@ -0,0 +1,28 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Comma-delimited list of one or more files containing PRRTE and PMIx
|
||||
MCA params for tuning DVM and/or application operations. Parameters in
|
||||
the file will be treated as *generic* parameters and subject to the
|
||||
translation rules/uncertainties. See ``--help mca`` for more
|
||||
information.
|
||||
|
||||
Syntax in the file is:
|
||||
|
||||
.. code::
|
||||
|
||||
param = value
|
||||
|
||||
with one parameter and its associated value per line. Empty lines and
|
||||
lines beginning with the ``#`` character are ignored.
|
||||
@@ -0,0 +1,23 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Export an environment variable, optionally specifying a value. For
|
||||
example:
|
||||
|
||||
* ``-x foo`` exports the environment variable ``foo`` and takes its
|
||||
value from the current environment.
|
||||
* ``-x foo=bar`` exports the environment variable name ``foo`` and
|
||||
sets its value to ``bar`` in the started processes.
|
||||
* ``-x foo*`` exports all current environmental variables starting
|
||||
with ``foo``.
|
||||
@@ -0,0 +1,21 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Definition of "processor element"
|
||||
=================================
|
||||
|
||||
By default, PRRTE defines that a "processing element" is a processor
|
||||
core. However, if ``--use-hwthread-cpus`` is specified on the command
|
||||
line, then a "processing element" is a hardware thread.
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Definition of 'slot'
|
||||
====================
|
||||
|
||||
The term "slot" is used extensively in the rest of this documentation.
|
||||
A slot is an allocation unit for a process. The number of slots on a
|
||||
node indicate how many processes can potentially execute on that node.
|
||||
By default, PRRTE will allow one process per slot.
|
||||
|
||||
If PRRTE is not explicitly told how many slots are available on a node
|
||||
(e.g., if a hostfile is used and the number of slots is not specified
|
||||
for a given node), it will determine a maximum number of slots for
|
||||
that node in one of two ways:
|
||||
|
||||
#. Default behavior: By default, PRRTE will attempt to discover the
|
||||
number of processor cores on the node, and use that as the number
|
||||
of slots available.
|
||||
|
||||
#. When ``--use-hwthread-cpus`` is used: If ``--use-hwthread-cpus`` is
|
||||
specified on the command line, then PRRTE will attempt to discover
|
||||
the number of hardware threads on the node, and use that as the
|
||||
number of slots available.
|
||||
|
||||
This default behavior also occurs when specifying the ``--host``
|
||||
option with a single host. Thus, the command:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
shell$ prun --host node1 ./a.out
|
||||
|
||||
launches a number of processes equal to the number of cores on node
|
||||
``node1``, whereas:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
shell$ prun --host node1 --use-hwthread-cpus ./a.out
|
||||
|
||||
launches a number of processes equal to the number of hardware
|
||||
threads on ``node1``.
|
||||
|
||||
When PRRTE applications are invoked in an environment managed by a
|
||||
resource manager (e.g., inside of a Slurm job), and PRRTE was built
|
||||
with appropriate support for that resource manager, then PRRTE will
|
||||
be informed of the number of slots for each node by the resource
|
||||
manager. For example:
|
||||
|
||||
.. code:: sh
|
||||
|
||||
shell$ prun ./a.out
|
||||
|
||||
launches one process for every slot (on every node) as dictated by
|
||||
the resource manager job specification.
|
||||
|
||||
Also note that the one-process-per-slot restriction can be overridden
|
||||
in unmanaged environments (e.g., when using hostfiles without a
|
||||
resource manager) if oversubscription is enabled (by default, it is
|
||||
disabled). Most parallel applications and HPC environments do not
|
||||
oversubscribe; for simplicity, the majority of this documentation
|
||||
assumes that oversubscription is not enabled.
|
||||
|
||||
Slots are not hardware resources
|
||||
--------------------------------
|
||||
|
||||
Slots are frequently incorrectly conflated with hardware resources.
|
||||
It is important to realize that slots are an entirely different metric
|
||||
than the number (and type) of hardware resources available.
|
||||
|
||||
Here are some examples that may help illustrate the difference:
|
||||
|
||||
#. More processor cores than slots: Consider a resource manager job
|
||||
environment that tells PRRTE that there is a single node with 20
|
||||
processor cores and 2 slots available. By default, PRRTE will
|
||||
only let you run up to 2 processes.
|
||||
|
||||
Meaning: you run out of slots long before you run out of processor
|
||||
cores.
|
||||
|
||||
#. More slots than processor cores: Consider a hostfile with a single
|
||||
node listed with a ``slots=50`` qualification. The node has 20
|
||||
processor cores. By default, PRRTE will let you run up to 50
|
||||
processes.
|
||||
|
||||
Meaning: you can run many more processes than you have processor
|
||||
cores.
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Bind each process to its own core.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--bind-to core``.
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Display the allocation being used by this job.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--display alloc``.
|
||||
@@ -0,0 +1,21 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Display a detailed list (mostly intended for developers) of the
|
||||
allocation being used by this job.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--display alloc-devel``.
|
||||
@@ -0,0 +1,21 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Display a detailed process map (mostly intended for developers)
|
||||
just before launch.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--display map-devel``.
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Display the process map just before launch.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--display map``.
|
||||
@@ -0,0 +1,21 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Display the topology as part of the process map (mostly intended
|
||||
for developers) just before launch.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--display topo``.
|
||||
@@ -0,0 +1,31 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Syntax: ``--gmca <key> <value>``, where ``key`` is the parameter name
|
||||
and ``value`` is the parameter value. The ``g`` prefix indicates that
|
||||
this parameter is "global", and to be applied to *all* application
|
||||
contexts |mdash| not just the one in which the directive appears.
|
||||
|
||||
Pass generic MCA parameters |mdash| i.e., parameters whose project
|
||||
affiliation must be determined by PRRTE based on matching the name of
|
||||
the parameter with defined values from various projects that PRRTE
|
||||
knows about.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This translation can be incomplete (e.g., if known project adds or
|
||||
changes parameters) |mdash| thus, it is strongly recommended that
|
||||
users use project-specific parameters such as ``--gprtemca`` or
|
||||
``--gpmixmca``.
|
||||
@@ -0,0 +1,29 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Syntax: ``--mca <key> <value>``, where ``key`` is the parameter name
|
||||
and ``value`` is the parameter value.
|
||||
|
||||
Pass generic MCA parameters |mdash| i.e., parameters whose project
|
||||
affiliation must be determined by PRRTE based on matching the name of
|
||||
the parameter with defined values from various projects that PRRTE
|
||||
knows about.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This translation can be incomplete (e.g., if a project adds or
|
||||
changes parameters) |mdash| thus, it is strongly recommended that
|
||||
users use project-specific parameters such as ``--prtemca`` or
|
||||
``--pmixmca``.
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Merge stderr to stdout for each process.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--output merge``
|
||||
@@ -0,0 +1,26 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Redirect output from application processes into
|
||||
``filename/job/rank/std[out,err,diag]``. A relative path value will be
|
||||
converted to an absolute path. The directory name may include a colon
|
||||
followed by a comma-delimited list of optional case-insensitive
|
||||
directives. Supported directives currently include ``NOJOBID`` (do not
|
||||
include a job-id directory level) and ``NOCOPY`` (do not copy the
|
||||
output to the stdout/err streams).
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--output dir=<path>``.
|
||||
@@ -0,0 +1,25 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Redirect output from application processes into ``filename.rank``. A
|
||||
relative path value will be converted to an absolute path. The
|
||||
directory name may include a colon followed by a comma-delimited list
|
||||
of optional case-insensitive directives. Supported directives
|
||||
currently include ``NOCOPY`` (do not copy the output to the stdout/err
|
||||
streams).
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--output file=<path>``
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Display process bindings to stderr.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--display bindings``.
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Tag all output with ``[job,rank]``.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--output``.
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Timestamp all application process output.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--output timestamp``.
|
||||
@@ -0,0 +1,20 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Provide all output in XML format.
|
||||
|
||||
.. admonition:: Deprecated
|
||||
:class: warning
|
||||
|
||||
This option is deprecated. Please use ``--output``.
|
||||
@@ -0,0 +1,48 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Hostfiles (sometimes called "machine files") are a combination of two
|
||||
things:
|
||||
|
||||
#. A listing of hosts on which to launch processes.
|
||||
#. Optionally, limit the number of processes which can be launched on
|
||||
each host.
|
||||
|
||||
Hostfile syntax consists of one node name on each line, optionally
|
||||
including a designated number of "slots":
|
||||
|
||||
.. code:: sh
|
||||
|
||||
# This is a comment line, and will be ignored
|
||||
node01 slots=10
|
||||
node13 slots=5
|
||||
|
||||
node15
|
||||
node16
|
||||
node17 slots=3
|
||||
...
|
||||
|
||||
Blank lines and lines beginning with a ``#`` are ignored.
|
||||
|
||||
A "slot" is the PRRTE term for an allocatable unit where we can launch
|
||||
a process. See the section on definition of the term ``slot`` for a
|
||||
longer description of slots.
|
||||
|
||||
In the absence of the ``slot`` parameter, PRRTE will assign either the
|
||||
number of slots to be the number of CPUs detected on the node or the
|
||||
resource manager-assigned value if operating in the presence of an
|
||||
RM.
|
||||
|
||||
.. important:: If using a resource manager, the user-specified number
|
||||
of slots is capped by the RM-assigned value.
|
||||
@@ -0,0 +1,47 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Listing Hosts on the Command Line
|
||||
=================================
|
||||
|
||||
Many PRRTE commands accept the ``--host`` CLI parameter.
|
||||
``--host`` accepts a comma-delimited list of tokens of the form:
|
||||
|
||||
.. code::
|
||||
|
||||
host[:slots]
|
||||
|
||||
The ``host`` token can be either:
|
||||
|
||||
* A name that resolves to an IP address, or
|
||||
* An IP address
|
||||
|
||||
.. note:: The names and/or IP addresses of hosts are *only* used for
|
||||
identifying the target host on which to launch. They are
|
||||
*not* used for determining which network interfaces are used
|
||||
by applications (e.g., MPI or other network-based
|
||||
applications).
|
||||
|
||||
For network-based applications, consult their documentation
|
||||
for how to specify which network interfaces are used.
|
||||
|
||||
The optional integer ``:slots`` parameter tells PRRTE the maximum
|
||||
number of slots to use on that host (see this section on definition
|
||||
of the term ``slot`` for a description of what a "slot" is).
|
||||
|
||||
For example:
|
||||
|
||||
.. code::
|
||||
|
||||
prterun --host node1:10,node2,node3:5 ...
|
||||
@@ -0,0 +1,127 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Relative host indexing
|
||||
======================
|
||||
|
||||
Hostfile and ``--host`` specifications can also be made using relative
|
||||
indexing. This allows a user to stipulate which hosts are to be used
|
||||
for a given app context without specifying the particular host name,
|
||||
but rather its relative position in the allocation.
|
||||
|
||||
This can probably best be understood through consideration of a few
|
||||
examples. Consider the case where a DVM is comprised of a set of nodes
|
||||
named ``foo1``, ``foo2``, ``foo3``, ``foo4``. The user wants the first
|
||||
app context to have exclusive use of the first two nodes, and a second
|
||||
app context to use the last two nodes. Of course, the user could
|
||||
printout the allocation to find the names of the nodes allocated to
|
||||
them and then use ``--host`` to specify this layout, but this is
|
||||
cumbersome and would require hand-manipulation for every invocation.
|
||||
|
||||
A simpler method is to utilize PRRTE's relative indexing capability to
|
||||
specify the desired layout. In this case, a command line containing:
|
||||
|
||||
.. code::
|
||||
|
||||
--host +n1,+n2 ./app1 : --host +n3,+n4 ./app2
|
||||
|
||||
would provide the desired pattern. The ``+`` syntax indicates that the
|
||||
information is being provided as a relative index into the existing
|
||||
allocation. Two methods of relative indexing are supported:
|
||||
|
||||
* ``+n#``: A relative index into the allocation referencing the ``#``
|
||||
node. PRRTE will substitute the ``#`` node in the allocation
|
||||
|
||||
* ``+e[:#]``: A request for ``#`` empty nodes |mdash| i.e., PRRTE is
|
||||
to substitute this reference with nodes that have not yet been used
|
||||
by any other app_context. If the ``:#`` is not provided, PRRTE will
|
||||
substitute the reference with all empty nodes. Note that PRRTE does
|
||||
track the empty nodes that have been assigned in this manner, so
|
||||
multiple uses of this option will result in assignment of unique
|
||||
nodes up to the limit of the available empty nodes. Requests for
|
||||
more empty nodes than are available will generate an error.
|
||||
|
||||
Relative indexing can be combined with absolute naming of hosts in any
|
||||
arbitrary manner, and can be used in hostfiles as well as with the
|
||||
``--host`` command line option. In addition, any slot specification
|
||||
provided in hostfiles will be respected |mdash| thus, a user can
|
||||
specify that only a certain number of slots from a relative indexed
|
||||
host are to be used for a given app context.
|
||||
|
||||
Another example may help illustrate this point. Consider the case
|
||||
where the user has a hostfile containing:
|
||||
|
||||
.. code::
|
||||
|
||||
dummy1 slots=4
|
||||
dummy2 slots=4
|
||||
dummy3 slots=4
|
||||
dummy4 slots=4
|
||||
dummy5 slots=4
|
||||
|
||||
This may, for example, be a hostfile that describes a set of
|
||||
commonly-used resources that the user wishes to execute applications
|
||||
against. For this particular application, the user plans to map
|
||||
byslot, and wants the first two ranks to be on the second node of any
|
||||
allocation, the next ranks to land on an empty node, have one rank
|
||||
specifically on ``dummy4``, the next rank to be on the second node of the
|
||||
allocation again, and finally any remaining ranks to be on whatever
|
||||
empty nodes are left. To accomplish this, the user provides a hostfile
|
||||
of:
|
||||
|
||||
.. code::
|
||||
|
||||
+n2 slots=2
|
||||
+e:1
|
||||
dummy4 slots=1
|
||||
+n2
|
||||
+e
|
||||
|
||||
The user can now use this information in combination with PRRTE's
|
||||
sequential mapper to obtain their specific layout:
|
||||
|
||||
.. code::
|
||||
|
||||
<launcher> --hostfile dummyhosts --hostfile mylayout --prtemca rmaps seq ./my_app
|
||||
|
||||
which will result in:
|
||||
|
||||
.. code::
|
||||
|
||||
rank0 being mapped to dummy3
|
||||
rank1 to dummy1 as the first empty node
|
||||
rank2 to dummy4
|
||||
rank3 to dummy3
|
||||
rank4 to dummy2 and rank5 to dummy5 as the last remaining unused nodes
|
||||
|
||||
Note that the sequential mapper ignores the number of slots arguments
|
||||
as it only maps one rank at a time to each node in the list.
|
||||
|
||||
If the default round-robin mapper had been used, then the mapping
|
||||
would have resulted in:
|
||||
|
||||
* ranks 0 and 1 being mapped to dummy3 since two slots were specified
|
||||
* ranks 2-5 on dummy1 as the first empty node, which has four slots
|
||||
* rank6 on dummy4 since the hostfile specifies only a single slot from
|
||||
that node is to be used
|
||||
* ranks 7 and 8 on dummy3 since only two slots remain available
|
||||
* ranks 9-12 on dummy2 since it is the next available empty node and
|
||||
has four slots
|
||||
* ranks 13-16 on dummy5 since it is the last remaining unused node and
|
||||
has four slots
|
||||
|
||||
Thus, the use of relative indexing can allow for complex mappings to
|
||||
be ported across allocations, including those obtained from automated
|
||||
resource managers, without the need for manual manipulation of scripts
|
||||
and/or command lines.
|
||||
@@ -0,0 +1,23 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Resource Manager-Provided Hosts
|
||||
===============================
|
||||
|
||||
When launching under a Resource Manager (RM), the RM usually
|
||||
picks which hosts |mdash| and how many processes can be launched on
|
||||
each host |mdash| on a per-job basis.
|
||||
|
||||
The RM will communicate this information to PRRTE directly; users can
|
||||
simply omit specifying hosts or numbers of processes.
|
||||
@@ -0,0 +1,155 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Deprecated options
|
||||
==================
|
||||
|
||||
These deprecated options will be removed in a future release.
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 20 20 30
|
||||
|
||||
* - Deprecated Option
|
||||
- Replacement
|
||||
- Description
|
||||
|
||||
* - ``--bind-to-core``
|
||||
- ``--bind-to core``
|
||||
- Bind processes to cores
|
||||
|
||||
|
||||
* - ``--bind-to-socket``
|
||||
- ``--bind-to package``
|
||||
- Bind processes to processor sockets
|
||||
|
||||
* - ``--bycore``
|
||||
- ``--map-by core``
|
||||
- Map processes by core
|
||||
|
||||
* - ``--bynode``
|
||||
- ``--map-by node``
|
||||
- Launch processes one per node, cycling by node in a round-robin
|
||||
fashion. This spreads processes evenly among nodes and assigns
|
||||
ranks in a round-robin, "by node" manner.
|
||||
|
||||
* - ``--byslot``
|
||||
- ``--map-by slot``
|
||||
- Map and rank processes round-robin by slot
|
||||
|
||||
* - ``--cpus-per-proc <#perproc>``
|
||||
- `--map-by <obj>:PE=<#perproc>``
|
||||
- Bind each process to the specified number of CPUs
|
||||
|
||||
* - ``--cpus-per-rank <#perrank>``
|
||||
- ``--map-by <obj>:PE=<#perrank>``
|
||||
- Alias for ``--cpus-per-proc``
|
||||
|
||||
* - ``--display-allocation``
|
||||
- ``--display ALLOC``
|
||||
- Display the detected resource allocation
|
||||
|
||||
* - ``-display-devel-map``
|
||||
- ``--display MAP-DEVEL``
|
||||
- Display a detailed process map (mostly intended for developers)
|
||||
just before launch.
|
||||
|
||||
* - ``--display-map``
|
||||
- ``--display MAP``
|
||||
- Display a table showing the mapped location of each process
|
||||
prior to launch.
|
||||
|
||||
* - ``--display-topo``
|
||||
- ``--display TOPO``
|
||||
- Display the topology as part of the process map (mostly
|
||||
intended for developers) just before launch.
|
||||
|
||||
* - ``--do-not-launch``
|
||||
- ``--map-by :DONOTLAUNCH``
|
||||
- Perform all necessary operations to prepare to launch the
|
||||
application, but do not actually launch it (usually used to
|
||||
test mapping patterns).
|
||||
|
||||
* - ``--do-not-resolve``
|
||||
- ``--map-by :DONOTRESOLVE``
|
||||
- Do not attempt to resolve interfaces |mdash| usually used to
|
||||
determine proposed process placement/binding prior to obtaining
|
||||
an allocation.
|
||||
|
||||
* - ``-N <num>``
|
||||
- ``--map-by prr:<num>:node``
|
||||
- Launch ``num`` processes per node on all allocated nodes
|
||||
|
||||
* - ``--nolocal``
|
||||
- ``--map-by :NOLOCAL``
|
||||
- Do not run any copies of the launched application on the same
|
||||
node as ``prun`` is running. This option will override listing
|
||||
the ``localhost`` with ``--host`` or any other host-specifying
|
||||
mechanism.
|
||||
|
||||
* - ``--nooversubscribe``
|
||||
- ``--map-by :NOOVERSUBSCRIBE``
|
||||
- Do not oversubscribe any nodes; error (without starting any
|
||||
processes) if the requested number of processes would cause
|
||||
oversubscription. This option implicitly sets "max_slots" equal
|
||||
to the "slots" value for each node. (Enabled by default).
|
||||
|
||||
* - ``--npernode <#pernode>``
|
||||
- ``--map-by ppr:<#pernode>:node``
|
||||
- On each node, launch this many processes
|
||||
|
||||
* - ``--npersocket <#persocket>``
|
||||
- ``--map-by ppr:<#perpackage>:package``
|
||||
- On each node, launch this many processes times the number of
|
||||
processor sockets on the node. The ``--npersocket`` option also
|
||||
turns on the ``--bind-to socket`` option. The term ``socket``
|
||||
has been globally replaced with ``package``.
|
||||
|
||||
* - ``--oversubscribe``
|
||||
- ``--map-by :OVERSUBSCRIBE``
|
||||
- Nodes are allowed to be oversubscribed, even on a managed
|
||||
system, and overloading of processing elements.
|
||||
|
||||
* - ``--pernode``
|
||||
- ``--map-by ppr:1:node``
|
||||
- On each node, launch one process
|
||||
|
||||
* - ``--ppr``
|
||||
- `--map-by ppr:<list>``
|
||||
- Comma-separated list of number of processes on a given resource type
|
||||
[default: ``none``].
|
||||
|
||||
* - ``--rankfile <FILENAME>``
|
||||
- ``--map-by rankfile:FILE=<FILENAME>``
|
||||
- Use a rankfile for mapping/ranking/binding
|
||||
|
||||
* - ``--report-bindings``
|
||||
- ``--display BINDINGS``
|
||||
- Report any bindings for launched processes
|
||||
|
||||
* - ``--tag-output``
|
||||
- ``--output TAG``
|
||||
- Tag all output with ``[job,rank]``
|
||||
|
||||
* - ``--timestamp-output``
|
||||
- ``--output TIMESTAMP``
|
||||
- Timestamp all application process output
|
||||
|
||||
* - ``--use-hwthread-cpus``
|
||||
- ``--map-by :HWTCPUS``
|
||||
- Use hardware threads as independent CPUs
|
||||
|
||||
* - ``--xml``
|
||||
- ``--output XML``
|
||||
- Provide all output in XML format
|
||||
@@ -0,0 +1,91 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Diagnostics
|
||||
===========
|
||||
|
||||
PRRTE provides various diagnostic reports that aid the user in
|
||||
verifying and tuning the mapping/ranking/binding for a specific job.
|
||||
|
||||
The ``:REPORT`` qualifier to the ``--bind-to`` command line option can
|
||||
be used to report process bindings.
|
||||
|
||||
As an example, consider a node with:
|
||||
|
||||
* 2 processor packages,
|
||||
* 4 cores per package, and
|
||||
* 8 hardware threads per core.
|
||||
|
||||
In each of the examples below the binding is reported in a human readable
|
||||
format.
|
||||
|
||||
.. code::
|
||||
|
||||
$ prun --np 4 --map-by core --bind-to core:REPORT ./a.out
|
||||
[node01:103137] MCW rank 0 bound to package[0][core:0]
|
||||
[node01:103137] MCW rank 1 bound to package[0][core:1]
|
||||
[node01:103137] MCW rank 2 bound to package[0][core:2]
|
||||
[node01:103137] MCW rank 3 bound to package[0][core:3]
|
||||
|
||||
In the example above, processes are bound to successive cores on the
|
||||
first package.
|
||||
|
||||
.. code::
|
||||
|
||||
$ prun --np 4 --map-by package --bind-to package:REPORT ./a.out
|
||||
[node01:103115] MCW rank 0 bound to package[0][core:0-9]
|
||||
[node01:103115] MCW rank 1 bound to package[1][core:10-19]
|
||||
[node01:103115] MCW rank 2 bound to package[0][core:0-9]
|
||||
[node01:103115] MCW rank 3 bound to package[1][core:10-19]
|
||||
|
||||
In the example above, processes are bound to all cores on successive
|
||||
packages in a round-robin fashion.
|
||||
|
||||
.. code::
|
||||
|
||||
$ prun --np 4 --map-by package:PE=2 --bind-to core:REPORT ./a.out
|
||||
[node01:103328] MCW rank 0 bound to package[0][core:0-1]
|
||||
[node01:103328] MCW rank 1 bound to package[1][core:10-11]
|
||||
[node01:103328] MCW rank 2 bound to package[0][core:2-3]
|
||||
[node01:103328] MCW rank 3 bound to package[1][core:12-13]
|
||||
|
||||
The example above shows us that 2 cores have been bound per process.
|
||||
The ``:PE=2`` qualifier states that 2 CPUs underneath the package
|
||||
(which would be cores in this case) are mapped to each process.
|
||||
|
||||
.. code::
|
||||
|
||||
$ prun --np 4 --map-by core:PE=2:HWTCPUS --bind-to :REPORT hostname
|
||||
[node01:103506] MCW rank 0 bound to package[0][hwt:0-1]
|
||||
[node01:103506] MCW rank 1 bound to package[0][hwt:8-9]
|
||||
[node01:103506] MCW rank 2 bound to package[0][hwt:16-17]
|
||||
[node01:103506] MCW rank 3 bound to package[0][hwt:24-25]
|
||||
|
||||
The example above shows us that 2 hardware threads have been bound per
|
||||
process. In this case ``prun`` is directing the DVM to map by
|
||||
hardware threads since we used the ``:HWTCPUS`` qualifier. Without
|
||||
that qualifier this command would return an error since by default the
|
||||
DVM will not map to resources smaller than a core. The ``:PE=2``
|
||||
qualifier states that 2 processing elements underneath the core (which
|
||||
would be hardware threads in this case) are mapped to each process.
|
||||
|
||||
.. code::
|
||||
|
||||
$ prun --np 4 --bind-to none:REPORT hostname
|
||||
[node01:107126] MCW rank 0 is not bound (or bound to all available processors)
|
||||
[node01:107126] MCW rank 1 is not bound (or bound to all available processors)
|
||||
[node01:107126] MCW rank 2 is not bound (or bound to all available processors)
|
||||
[node01:107126] MCW rank 3 is not bound (or bound to all available processors)
|
||||
|
||||
Binding is turned off in the above example, as reported.
|
||||
@@ -0,0 +1,434 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Examples
|
||||
========
|
||||
|
||||
Listed here are the subset of command line options that will be used
|
||||
in the process mapping/ranking/binding examples below.
|
||||
|
||||
Specifying Host Nodes
|
||||
---------------------
|
||||
|
||||
Use one of the following options to specify which hosts (nodes) within
|
||||
the PRRTE DVM environment to run on.
|
||||
|
||||
.. code::
|
||||
|
||||
--host <host1,host2,...,hostN>
|
||||
|
||||
# or
|
||||
|
||||
--host <host1:X,host2:Y,...,hostN:Z>
|
||||
|
||||
* List of hosts on which to invoke processes. After each hostname a
|
||||
colon (``:``) followed by a positive integer can be used to specify
|
||||
the number of slots on that host (``:X``, ``:Y``, and ``:Z``). The
|
||||
default is ``1``.
|
||||
|
||||
.. code::
|
||||
|
||||
--hostfile <hostfile>
|
||||
|
||||
* Provide a hostfile to use.
|
||||
|
||||
Process Mapping / Ranking / Binding Options
|
||||
-------------------------------------------
|
||||
|
||||
* ``-c #``, ``-n #``, ``--n #``, ``--np <#>``: Run this many copies of
|
||||
the program on the given nodes. This option indicates that the
|
||||
specified file is an executable program and not an application
|
||||
context. If no value is provided for the number of copies to execute
|
||||
(i.e., neither the ``-np`` nor its synonyms are provided on the
|
||||
command line), ``prun`` will automatically execute a copy of the
|
||||
program on each process slot (see below for description of a
|
||||
"process slot"). This feature, however, can only be used in the SPMD
|
||||
model and will return an error (without beginning execution of the
|
||||
application) otherwise.
|
||||
|
||||
.. note:: These options specify the number of processes to launch.
|
||||
None of the options imply a particular binding policy
|
||||
|mdash| e.g., requesting ``N`` processes for each package
|
||||
does not imply that the processes will be bound to the
|
||||
package.
|
||||
|
||||
* ``--map-by <object>``: Map to the specified object. Supported
|
||||
objects include:
|
||||
|
||||
* ``slot``
|
||||
* ``hwthread``
|
||||
* ``core`` (default)
|
||||
* ``l1cache``
|
||||
* ``l2cache``
|
||||
* ``l3cache``
|
||||
* ``numa``
|
||||
* ``package``
|
||||
* ``node``
|
||||
* ``seq``
|
||||
* ``ppr``
|
||||
* ``rankfile``
|
||||
* ``pe-list``
|
||||
|
||||
Any object can include qualifiers by adding a colon (``:``) and any
|
||||
colon-delimited combination of one or more of the following to the
|
||||
``--map-by`` options:
|
||||
|
||||
* ``PE=n`` bind ``n`` processing elements to each process (can not
|
||||
be used in combination with rankfile or pe-list directives)
|
||||
|
||||
.. error:: JMS Several of the options below refer to ``pe-list``.
|
||||
Is this option supposed to be ``PE-LIST=n``, not
|
||||
``PE=n``?
|
||||
|
||||
* ``SPAN`` load balance the processes across the allocation (cannot
|
||||
be used in combination with ``slot``, ``node``, ``seq``, ``ppr``,
|
||||
``rankfile``, or ``pe-list`` directives)
|
||||
|
||||
* ``OVERSUBSCRIBE`` allow more processes on a node than processing
|
||||
elements
|
||||
|
||||
* ``NOOVERSUBSCRIBE`` means ``!OVERSUBSCRIBE``
|
||||
|
||||
* ``NOLOCAL`` do not launch processes on the same node as ``prun``
|
||||
|
||||
* ``HWTCPUS`` use hardware threads as CPU slots
|
||||
|
||||
* ``CORECPUS`` use cores as CPU slots (default)
|
||||
|
||||
* ``INHERIT`` indicates that a child job (i.e., one spawned from
|
||||
within an application) shall inherit the placement policies of the
|
||||
parent job that spawned it.
|
||||
|
||||
* ``NOINHERIT`` means ``!INHERIT``
|
||||
|
||||
* ``FILE=<path>`` (path to file containing sequential or rankfile
|
||||
entries).
|
||||
|
||||
* ``ORDERED`` only applies to the PE-LIST option to indicate that
|
||||
procs are to be bound to each of the specified CPUs in the order
|
||||
in which they are assigned (i.e., the first proc on a node shall
|
||||
be bound to the first CPU in the list, the second proc shall be
|
||||
bound to the second CPU, etc.)
|
||||
|
||||
``ppr`` policy example: ``--map-by ppr:N:<object>`` will launch
|
||||
``N`` times the number of objects of the specified type on each
|
||||
node.
|
||||
|
||||
.. note:: Directives and qualifiers are case-insensitive and can be
|
||||
shortened to the minimum number of characters to uniquely
|
||||
identify them. Thus, ``L1CACHE`` can be given as
|
||||
``l1cache`` or simply as ``L1``.
|
||||
|
||||
* ``--rank-by <object>``: This assigns ranks in round-robin fashion
|
||||
according to the specified object. The default follows the mapping
|
||||
pattern. Supported rank-by objects include:
|
||||
|
||||
* ``slot``
|
||||
* ``node``
|
||||
* ``fill``
|
||||
* ``span``
|
||||
|
||||
There are no qualifiers for the ``--rank-by`` directive.
|
||||
|
||||
* ``--bind-to <object>``: This binds processes to the specified
|
||||
object. See defaults in Quick Summary. Supported bind-to objects
|
||||
include:
|
||||
|
||||
* ``none``
|
||||
* ``hwthread``
|
||||
* ``core``
|
||||
* ``l1cache``
|
||||
* ``l2cache``
|
||||
* ``l3cache``
|
||||
* ``numa``
|
||||
* ``package``
|
||||
|
||||
Any object can include qualifiers by adding a colon (``:``) and any
|
||||
colon-delimited combination of one or more of the following to the
|
||||
``--bind-to`` options:
|
||||
|
||||
* ``overload-allowed`` allows for binding more than one process in
|
||||
relation to a CPU
|
||||
|
||||
* ``if-supported`` if binding to that object is supported on this
|
||||
system.
|
||||
|
||||
Specifying Host Nodes
|
||||
---------------------
|
||||
|
||||
Host nodes can be identified on the command line with the ``--host``
|
||||
option or in a hostfile.
|
||||
|
||||
For example, assuming no other resource manager or scheduler is
|
||||
involved:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --host aa,aa,bb ./a.out
|
||||
|
||||
This launches two processes on node ``aa`` and one on ``bb``.
|
||||
|
||||
.. code::
|
||||
|
||||
prun --host aa ./a.out
|
||||
|
||||
This launches one process on node ``aa``.
|
||||
|
||||
.. code::
|
||||
|
||||
prun --host aa:5 ./a.out
|
||||
|
||||
This launches five processes on node ``aa``.
|
||||
|
||||
Or, consider the hostfile:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
aa slots=2
|
||||
bb slots=2
|
||||
cc slots=2
|
||||
|
||||
Here, we list both the host names (``aa``, ``bb``, and ``cc``) but
|
||||
also how many "slots" there are for each. Slots indicate how many
|
||||
processes can potentially execute on a node. For best performance, the
|
||||
number of slots may be chosen to be the number of cores on the node or
|
||||
the number of processor sockets.
|
||||
|
||||
If the hostfile does not provide slots information, the PRRTE DVM will
|
||||
attempt to discover the number of cores (or hwthreads, if the
|
||||
``:HWTCPUS`` qualifier to the ``--map-by`` option is set) and set the
|
||||
number of slots to that value.
|
||||
|
||||
Examples using the hostfile above with and without the ``--host``
|
||||
option:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile ./a.out
|
||||
|
||||
This will launch two processes on each of the three nodes.
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --host aa ./a.out
|
||||
|
||||
This will launch two processes, both on node ``aa``.
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --host dd ./a.out
|
||||
|
||||
This will find no hosts to run on and abort with an error. That is, the
|
||||
specified host ``dd`` is not in the specified hostfile.
|
||||
|
||||
When running under resource managers (e.g., SLURM, Torque, etc.), PRTE
|
||||
will obtain both the hostnames and the number of slots directly from
|
||||
the resource manger. The behavior of ``--host`` in that environment
|
||||
will behave the same as if a hostfile was provided (since it is
|
||||
provided by the resource manager).
|
||||
|
||||
|
||||
Specifying Number of Processes
|
||||
------------------------------
|
||||
|
||||
As we have just seen, the number of processes to run can be set using
|
||||
the hostfile. Other mechanisms exist.
|
||||
|
||||
The number of processes launched can be specified as a multiple of the
|
||||
number of nodes or processor sockets available. Consider the hostfile
|
||||
below for the examples that follow.
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
aa
|
||||
bb
|
||||
|
||||
For example:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --map-by ppr:2:package ./a.out
|
||||
|
||||
This launches processes 0-3 on node ``aa`` and process 4-7 on node
|
||||
``bb``, where ``aa`` and ``bb`` are both dual-package nodes. The
|
||||
``--map-by ppr:2:package`` option also turns on the ``--bind-to
|
||||
package`` option, which is discussed in a later section.
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --map-by ppr:2:node ./a.out
|
||||
|
||||
This launches processes 0-1 on node ``aa`` and processes 2-3 on node
|
||||
``bb``.
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --map-by ppr:1:node ./a.out
|
||||
|
||||
This launches one process per host node.
|
||||
|
||||
Another alternative is to specify the number of processes with the
|
||||
``--np`` option. Consider now the hostfile:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
aa slots=4
|
||||
bb slots=4
|
||||
cc slots=4
|
||||
|
||||
With this hostfile:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --np 6 ./a.out
|
||||
|
||||
This will launch processes 0-3 on node ``aa`` and processes 4-5 on
|
||||
node ``bb``. The remaining slots in the hostfile will not be used
|
||||
since the ``-np`` option indicated that only 6 processes should be
|
||||
launched.
|
||||
|
||||
|
||||
Mapping Processes to Nodes Using Policies
|
||||
-----------------------------------------
|
||||
|
||||
The examples above illustrate the default mapping of process processes
|
||||
to nodes. This mapping can also be controlled with various
|
||||
``prun`` / ``prterun`` options that describe mapping policies.
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
aa slots=4
|
||||
bb slots=4
|
||||
cc slots=4
|
||||
|
||||
Consider the hostfile above, with ``--np 6``:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - Command
|
||||
- Ranks on ``aa``
|
||||
- Ranks on ``bb``
|
||||
- Ranks on ``cc``
|
||||
|
||||
* - ``prun``
|
||||
- 0 1 2 3
|
||||
- 4 5
|
||||
-
|
||||
|
||||
* - ``prun --map-by node``
|
||||
- 0 3
|
||||
- 1 4
|
||||
- 2 5
|
||||
|
||||
* - ``prun --map-by node:NOLOCAL``
|
||||
-
|
||||
- 0 2 4
|
||||
- 1 3 5
|
||||
|
||||
The ``--map-by node`` option will load balance the processes across
|
||||
the available nodes, numbering each process by node in a round-robin
|
||||
fashion.
|
||||
|
||||
The ``:NOLOCAL`` qualifier to ``--map-by`` prevents any processes from
|
||||
being mapped onto the local host (in this case node ``aa``). While
|
||||
``prun`` typically consumes few system resources, the ``:NOLOCAL``
|
||||
qualifier can be helpful for launching very large jobs where ``prun``
|
||||
may actually need to use noticeable amounts of memory and/or
|
||||
processing time.
|
||||
|
||||
Just as ``--np`` can specify fewer processes than there are slots, it
|
||||
can also oversubscribe the slots. For example, with the same hostfile:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --np 14 ./a.out
|
||||
|
||||
This will produce an error since the default ``:NOOVERSUBSCRIBE``
|
||||
qualifier to ``--map-by`` prevents oversubscription.
|
||||
|
||||
To oversubscribe the nodes you can use the ``:OVERSUBSCRIBE``
|
||||
qualifier to ``--map-by``:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --np 14 --map-by :OVERSUBSCRIBE ./a.out
|
||||
|
||||
This will launch processes 0-5 on node ``aa``, 6-9 on ``bb``, and
|
||||
10-13 on ``cc``.
|
||||
|
||||
Limits to oversubscription can also be specified in the hostfile
|
||||
itself with the ``max_slots`` field:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
aa slots=4 max_slots=4
|
||||
bb max_slots=8
|
||||
cc slots=4
|
||||
|
||||
The ``max_slots`` field specifies such a limit. When it does, the
|
||||
``slots`` value defaults to the limit. Now:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --hostfile myhostfile --np 14 --map-by :OVERSUBSCRIBE ./a.out
|
||||
|
||||
This causes the first 12 processes to be launched as before, but the
|
||||
remaining two processes will be forced onto node cc. The other two
|
||||
nodes are protected by the hostfile against oversubscription by this
|
||||
job.
|
||||
|
||||
Using the ``:NOOVERSUBSCRIBE`` qualifier to ``--map-by`` option can be
|
||||
helpful since the PRTE DVM currently does not get ``max_slots`` values
|
||||
from the resource manager.
|
||||
|
||||
Of course, ``--np`` can also be used with the ``--host`` option. For
|
||||
example,
|
||||
|
||||
.. code::
|
||||
|
||||
prun --host aa,bb --np 8 ./a.out
|
||||
|
||||
This will produce an error since the default ``:NOOVERSUBSCRIBE``
|
||||
qualifier to ``--map-by`` prevents oversubscription.
|
||||
|
||||
.. code::
|
||||
|
||||
prun --host aa,bb --np 8 --map-by :OVERSUBSCRIBE ./a.out
|
||||
|
||||
This launches 8 processes. Since only two hosts are specified, after
|
||||
the first two processes are mapped, one to ``aa`` and one to ``bb``,
|
||||
the remaining processes oversubscribe the specified hosts evenly.
|
||||
|
||||
.. code::
|
||||
|
||||
prun --host aa:2,bb:6 --np 8 ./a.out
|
||||
|
||||
This launches 8 processes. Processes 0-1 on node ``aa`` since it has 2
|
||||
slots and processes 2-7 on node ``bb`` since it has 6 slots.
|
||||
|
||||
And here is a MIMD example:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --host aa --np 1 hostname : --host bb,cc --np 2 uptime
|
||||
|
||||
This will launch process 0 running ``hostname`` on node ``aa`` and
|
||||
processes 1 and 2 each running ``uptime`` on nodes ``bb`` and ``cc``,
|
||||
respectively.
|
||||
@@ -0,0 +1,156 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Fundamentals
|
||||
============
|
||||
|
||||
The mapping of processes to nodes can be defined not just with general
|
||||
policies but also, if necessary, using arbitrary mappings that cannot
|
||||
be described by a simple policy. Supported directives, given on the
|
||||
command line via the ``--map-by`` option, include:
|
||||
|
||||
* ``SEQ``: (often accompanied by the ``file=<path>`` qualifier)
|
||||
assigns one process to each node specified in the file. The
|
||||
sequential file is to contain an entry for each desired process, one
|
||||
per line of the file.
|
||||
|
||||
* ``RANKFILE``: (often accompanied by the ``file=<path>`` qualifier)
|
||||
assigns one process to the node/resource specified in each entry of
|
||||
the file, one per line of the file.
|
||||
|
||||
For example, using the hostfile below:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
aa slots=4
|
||||
bb slots=4
|
||||
cc slots=4
|
||||
|
||||
The command below will launch three processes, one on each of nodes
|
||||
``aa``, ``bb``, and ``cc``, respectively. The slot counts don't
|
||||
matter; one process is launched per line on whatever node is listed on
|
||||
the line.
|
||||
|
||||
.. code::
|
||||
|
||||
$ prun --hostfile myhostfile --map-by seq ./a.out
|
||||
|
||||
Impact of the ranking option is best illustrated by considering the
|
||||
following hostfile and test cases where each node contains two
|
||||
packages (each package with two cores). Using the ``--map-by
|
||||
ppr:2:package`` option, we map two processes onto each package and
|
||||
utilize the ``--rank-by`` option as show below:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
aa
|
||||
bb
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - Command
|
||||
- Ranks on ``aa``
|
||||
- Ranks on ``bb``
|
||||
|
||||
* - ``--rank-by core``
|
||||
- 0 1 ! 2 3
|
||||
- 4 5 ! 6 7
|
||||
|
||||
* - ``--rank-by package``
|
||||
- 0 2 ! 1 3
|
||||
- 4 6 ! 5 7
|
||||
|
||||
* - ``--rank-by package:SPAN``
|
||||
- 0 4 ! 1 5
|
||||
- 2 6 ! 3 7
|
||||
|
||||
Ranking by slot provides the identical result as ranking by core in
|
||||
this case |mdash| a simple progression of ranks across each
|
||||
node. Ranking by package does a round-robin ranking across packages
|
||||
within each node until all processes have been assigned a rank, and
|
||||
then progresses to the next node. Adding the ``:SPAN`` qualifier to
|
||||
the ranking directive causes the ranking algorithm to treat the entire
|
||||
allocation as a single entity |mdash| thus, the process ranks are
|
||||
assigned across all packages before circling back around to the
|
||||
beginning.
|
||||
|
||||
The binding operation restricts the process to a subset of the CPU
|
||||
resources on the node.
|
||||
|
||||
The processors to be used for binding can be identified in terms of
|
||||
topological groupings |mdash| e.g., binding to an l3cache will bind
|
||||
each process to all processors within the scope of a single L3 cache
|
||||
within their assigned location. Thus, if a process is assigned by the
|
||||
mapper to a certain package, then a ``--bind-to l3cache`` directive
|
||||
will cause the process to be bound to the processors that share a
|
||||
single L3 cache within that package.
|
||||
|
||||
To help balance loads, the binding directive uses a round-robin method,
|
||||
binding a process to the first available specified object type within
|
||||
the object where the process was mapped. For example, consider the case
|
||||
where a job is mapped to the package level, and then bound to core. Each
|
||||
package will have multiple cores, so if multiple processes are mapped to
|
||||
a given package, the binding algorithm will assign each process located
|
||||
to a package to a unique core in a round-robin manner.
|
||||
|
||||
Binding can only be done to the mapped object or to a resource located
|
||||
within that object.
|
||||
|
||||
An object is considered completely consumed when the number of
|
||||
processes bound to it equals the number of CPUs within it. Unbound
|
||||
processes are not considered in this computation. Additional
|
||||
processes cannot be mapped to consumed objects unless the
|
||||
OVERLOAD qualifier is provided via the "--bind-to" command
|
||||
line option.
|
||||
|
||||
Default process mapping/ranking/binding policies can also be set with MCA
|
||||
parameters, overridden by the command line options when provided. MCA
|
||||
parameters can be set on the ``prte`` command line when starting the
|
||||
DVM (or in the ``prterun`` command line for a single-execution job), but
|
||||
also in a system or user ``mca-params.conf`` file or as environment
|
||||
variables, as described in the MCA section below. Some examples include:
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
|
||||
* - ``prun`` option
|
||||
- MCA parameter key
|
||||
- Value
|
||||
|
||||
* - ``--map-by core``
|
||||
- ``rmaps_default_mapping_policy``
|
||||
- ``core``
|
||||
|
||||
* - ``--map-by package``
|
||||
- ``rmaps_default_mapping_policy``
|
||||
- ``package``
|
||||
|
||||
* - ``--rank-by core``
|
||||
- ``rmaps_default_ranking_policy``
|
||||
- ``core``
|
||||
|
||||
* - ``--bind-to core``
|
||||
- ``hwloc_default_binding_policy``
|
||||
- ``core```
|
||||
|
||||
* - ``--bind-to package``
|
||||
- ``hwloc_default_binding_policy``
|
||||
- ``package``
|
||||
|
||||
* - ``--bind-to none``
|
||||
- ``hwloc_default_binding_policy``
|
||||
- ``none``
|
||||
@@ -0,0 +1,212 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Overloading and Oversubscribing
|
||||
===============================
|
||||
|
||||
This section explores the difference between the terms "overloading"
|
||||
and "oversubscribing". Users are often confused by the difference
|
||||
between these two scenarios. As such, this section provides a number
|
||||
of scenarios to help illustrate the differences.
|
||||
|
||||
* ``--map-by :OVERSUBSCRIBE`` allow more processes on a node than
|
||||
allocated
|
||||
|
||||
* ``--bind-to <object>:overload-allowed`` allows for binding more than
|
||||
one process in relation to a CPU
|
||||
|
||||
The important thing to remember with *oversubscribing* is that it can
|
||||
be defined separately from the actual number of CPUs on a node. This
|
||||
allows the mapper to place more or fewer processes per node than
|
||||
CPUs. By default, PRRTE uses cores to determine slots in the absence
|
||||
of such information provided in the hostfile or by the resource
|
||||
manager (except in the case of the ``--host`` as described in
|
||||
the section on that command line option.
|
||||
|
||||
The important thing to remember with *overloading* is that it is
|
||||
defined as binding more processes than CPUs. By default, PRRTE uses
|
||||
cores as a means of counting the number of CPUs. However, the user can
|
||||
adjust this. For example when using the ``:HWTCPUS`` qualifier to the
|
||||
``--map-by`` option PRRTE will use hardware threads as a means of
|
||||
counting the number of CPUs.
|
||||
|
||||
For the following examples consider a node with:
|
||||
|
||||
* 2 processor packages,
|
||||
* 10 cores per package, and
|
||||
* 8 hardware threads per core.
|
||||
|
||||
Consider the node from above with the hostfile below:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
node01 slots=32
|
||||
node02 slots=32
|
||||
|
||||
The ``slots`` token tells PRRTE that it can place up to 32 processes
|
||||
before *oversubscribing* the node.
|
||||
|
||||
If we run the following:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 34 --hostfile myhostfile --map-by core --bind-to core hostname
|
||||
|
||||
It will return an error at the binding time indicating an
|
||||
*overloading* scenario.
|
||||
|
||||
The mapping mechanism assigns 32 processes to ``node01`` matching the
|
||||
``slots`` specification in the hostfile. The binding mechanism will bind
|
||||
the first 20 processes to unique cores leaving it with 12 processes
|
||||
that it cannot bind without overloading one of the cores (putting more
|
||||
than one process on the core).
|
||||
|
||||
Using the ``overload-allowed`` qualifier to the ``--bind-to core``
|
||||
option tells PRRTE that it may assign more than one process to a core.
|
||||
|
||||
If we run the following:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 34 --hostfile myhostfile --map-by core --bind-to core:overload-allowed hostname
|
||||
|
||||
This will run correctly placing 32 processes on ``node01``, and 2
|
||||
processes on ``node02``. On ``node01`` two processes are bound to
|
||||
cores 0-11 accounting for the overloading of those cores.
|
||||
|
||||
Alternatively, we could use hardware threads to give binding a lower
|
||||
level CPU to bind to without overloading.
|
||||
|
||||
If we run the following:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 34 --hostfile myhostfile --map-by core:HWTCPUS --bind-to hwthread hostname
|
||||
|
||||
This will run correctly placing 32 processes on ``node01``, and 2
|
||||
processes on ``node02``. On ``node01`` two processes are mapped to
|
||||
cores 0-11 but bound to different hardware threads on those cores (the
|
||||
logical first and second hardware thread). Thus no hardware threads
|
||||
are overloaded at binding time.
|
||||
|
||||
In both of the examples above the node is not oversubscribed at
|
||||
mapping time because the hostfile set the oversubscription limit to
|
||||
``slots=32`` for each node. It is only after we exceed that limit that
|
||||
PRRTE will throw an oversubscription error.
|
||||
|
||||
Consider next if we ran the following:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 66 --hostfile myhostfile --map-by core:HWTCPUS --bind-to hwthread hostname
|
||||
|
||||
This will return an error at mapping time indicating an
|
||||
oversubscription scenario. The mapping mechanism will assign all of
|
||||
the available slots (64 across 2 nodes) and be left two processes to
|
||||
map. The only way to map those processes is to exceed the number of
|
||||
available slots putting the job into an oversubscription scenario.
|
||||
|
||||
You can force PRRTE to oversubscribe the nodes by using the
|
||||
``:OVERSUBSCRIBE`` qualifier to the ``--map-by`` option as seen in the
|
||||
example below:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 66 --hostfile myhostfile \
|
||||
--map-by core:HWTCPUS:OVERSUBSCRIBE --bind-to hwthread hostname
|
||||
|
||||
This will run correctly placing 34 processes on ``node01`` and 32 on
|
||||
``node02``. Each process is bound to a unique hardware thread.
|
||||
|
||||
Overloading vs. Oversubscription: Package Example
|
||||
-------------------------------------------------
|
||||
|
||||
Let's extend these examples by considering the package level.
|
||||
Consider the same node as before, but with the hostfile below:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
node01 slots=22
|
||||
node02 slots=22
|
||||
|
||||
The lowest level CPUs are "cores" and we have 20 total (10 per
|
||||
package).
|
||||
|
||||
If we run:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 20 --hostfile myhostfile --map-by package \
|
||||
--bind-to package:REPORT hostname
|
||||
|
||||
Then 10 processes are mapped to each package, and bound at the package
|
||||
level. This is not overloading since we have 10 CPUs (cores)
|
||||
available in the package at the hardware level.
|
||||
|
||||
However, if we run:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 21 --hostfile myhostfile --map-by package \
|
||||
--bind-to package:REPORT hostname
|
||||
|
||||
Then 11 processes are mapped to the first package and 10 to the second
|
||||
package. At binding time we have an overloading scenario because
|
||||
there are only 10 CPUs (cores) available in the package at the
|
||||
hardware level. So the first package is overloaded.
|
||||
|
||||
Overloading vs. Oversubscription: Hardware Threads Example
|
||||
----------------------------------------------------------
|
||||
|
||||
Similarly, if we consider hardware threads.
|
||||
|
||||
Consider the same node as before, but with the hostfile below:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myhostfile
|
||||
node01 slots=165
|
||||
node02 slots=165
|
||||
|
||||
The lowest level CPUs are "hwthreads" (because we are going to use the
|
||||
``:HWTCPUS`` qualifier) and we have 160 total (80 per package).
|
||||
|
||||
If we re-run (from the package example) and add the ``:HWTCPUS``
|
||||
qualifier:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 21 --hostfile myhostfile --map-by package:HWTCPUS \
|
||||
--bind-to package:REPORT hostname
|
||||
|
||||
Without the ``:HWTCPUS`` qualifier this would be overloading (as we
|
||||
saw previously). The mapper places 11 processes on the first package
|
||||
and 10 to the second package. The processes are still bound to the
|
||||
package level. However, with the ``:HWTCPUS`` qualifier, it is not
|
||||
overloading since we have 80 CPUs (hwthreads) available in the package
|
||||
at the hardware level.
|
||||
|
||||
Alternatively, if we run:
|
||||
|
||||
.. code::
|
||||
|
||||
prun --np 161 --hostfile myhostfile --map-by package:HWTCPUS \
|
||||
--bind-to package:REPORT hostname
|
||||
|
||||
Then 81 processes are mapped to the first package and 80 to the second
|
||||
package. At binding time we have an overloading scenario because
|
||||
there are only 80 CPUs (hwthreads) available in the package at the
|
||||
hardware level. So the first package is overloaded.
|
||||
@@ -0,0 +1,85 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Rankfiles
|
||||
=========
|
||||
|
||||
Another way to specify arbitrary mappings is with a rankfile, which
|
||||
gives you detailed control over process binding as well.
|
||||
|
||||
Rankfiles are text files that specify detailed information about how
|
||||
individual processes should be mapped to nodes, and to which
|
||||
processor(s) they should be bound. Each line of a rankfile specifies
|
||||
the location of one process. The general form of each line in the
|
||||
rankfile is:
|
||||
|
||||
.. code::
|
||||
|
||||
rank <N>=<hostname> slot=<slot list>
|
||||
|
||||
For example:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myrankfile
|
||||
rank 0=aa slot=10-12
|
||||
rank 1=bb slot=0,1,4
|
||||
rank 2=cc slot=1-2
|
||||
$ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
|
||||
|
||||
Means that:
|
||||
|
||||
* Rank 0 runs on node aa, bound to logical cores 10-12.
|
||||
* Rank 1 runs on node bb, bound to logical cores 0, 1, and 4.
|
||||
* Rank 2 runs on node cc, bound to logical cores 1 and 2.
|
||||
|
||||
Similarly:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myrankfile
|
||||
rank 0=aa slot=1:0-2
|
||||
rank 1=bb slot=0:0,1,4
|
||||
rank 2=cc slot=1-2
|
||||
$ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
|
||||
|
||||
Means that:
|
||||
|
||||
* Rank 0 runs on node aa, bound to logical package 1, cores 10-12 (the
|
||||
0th through 2nd cores on that package).
|
||||
* Rank 1 runs on node bb, bound to logical package 0, cores 0, 1,
|
||||
and 4.
|
||||
* Rank 2 runs on node cc, bound to logical cores 1 and 2.
|
||||
|
||||
The hostnames listed above are "absolute," meaning that actual
|
||||
resolvable hostnames are specified. However, hostnames can also be
|
||||
specified as "relative," meaning that they are specified in relation
|
||||
to an externally-specified list of hostnames (e.g., by ``prun``'s
|
||||
``--host`` argument, a hostfile, or a job scheduler).
|
||||
|
||||
The "relative" specification is of the form "``+n<X>``", where ``X``
|
||||
is an integer specifying the Xth hostname in the set of all available
|
||||
hostnames, indexed from 0. For example:
|
||||
|
||||
.. code::
|
||||
|
||||
$ cat myrankfile
|
||||
rank 0=+n0 slot=10-12
|
||||
rank 1=+n1 slot=0,1,4
|
||||
rank 2=+n2 slot=1-2
|
||||
$ prun --host aa,bb,cc,dd --map-by rankfile:FILE=myrankfile ./a.out
|
||||
|
||||
All package/core slot locations are be specified as *logical*
|
||||
indexes. You can use tools such as HWLOC's ``lstopo`` to find the
|
||||
logical indexes of packages and cores.
|
||||
@@ -0,0 +1,113 @@
|
||||
.. -*- rst -*-
|
||||
|
||||
Copyright (c) 2022-2023 Nanook Consulting. All rights reserved.
|
||||
Copyright (c) 2023 Jeffrey M. Squyres. All rights reserved.
|
||||
|
||||
$COPYRIGHT$
|
||||
|
||||
Additional copyrights may follow
|
||||
|
||||
$HEADER$
|
||||
|
||||
.. The following line is included so that Sphinx won't complain
|
||||
about this file not being directly included in some toctree
|
||||
|
||||
Overview
|
||||
========
|
||||
|
||||
PRRTE provides a set of three controls for assigning process
|
||||
locations and ranks:
|
||||
|
||||
#. Mapping: Assigns a default location to each process
|
||||
#. Ranking: Assigns a unique integer rank value to each process
|
||||
#. Binding: Constrains each process to run on specific processors
|
||||
|
||||
This section provides an overview of these three controls. Unless
|
||||
otherwise this behavior is shared by ``prun(1)`` (working with a PRRTE
|
||||
DVM), and ``prterun(1)``. More detail about PRRTE process placement is
|
||||
available in the following sections (using ``--help
|
||||
placement-<section>``):
|
||||
|
||||
* ``examples``: some examples of the interactions between mapping,
|
||||
ranking, and binding options.
|
||||
|
||||
* ``fundamentals``: provides deeper insight into PRRTE's mapping,
|
||||
ranking, and binding options.
|
||||
|
||||
* ``limits``: explains the difference between *overloading* and
|
||||
*oversubscribing* resources.
|
||||
|
||||
* ``diagnostics``: describes options for obtaining various diagnostic
|
||||
reports that aid the user in verifying and tuning the placement for
|
||||
a specific job.
|
||||
|
||||
* ``rankfiles``: explains the format and use of the rankfile mapper
|
||||
for specifying arbitrary process placements.
|
||||
|
||||
* ``deprecated``: a list of deprecated options and their new
|
||||
equivalents.
|
||||
|
||||
* ``all``: outputs all the placement help except for the
|
||||
``deprecated`` section.
|
||||
|
||||
|
||||
Quick Summary
|
||||
-------------
|
||||
|
||||
The two binaries that most influence process layout are ``prte(1)``
|
||||
and ``prun(1)``. The ``prte(1)`` process discovers the allocation,
|
||||
establishes a Distributed Virtual Machine by starting a ``prted(1)``
|
||||
daemon on each node of the allocation, and defines the efault
|
||||
mapping/ranking/binding policies for all jobs. The ``prun(1)`` process
|
||||
defines the specific mapping/ranking/binding for a specific job. Most
|
||||
of the command line controls are targeted to ``prun(1)`` since each job
|
||||
has its own unique requirements.
|
||||
|
||||
``prterun(1)`` is just a wrapper around ``prte(1)`` for a single job
|
||||
PRRTE DVM. It is doing the job of both ``prte(1)`` and ``prun(1)``,
|
||||
and, as such, accepts the sum all of their command line arguments. Any
|
||||
example that uses ``prun(1)`` can substitute the use of ``prterun(1)``
|
||||
except where otherwise noted.
|
||||
|
||||
The ``prte(1)`` process attempts to automatically discover the nodes
|
||||
in the allocation by querying supported resource managers. If a
|
||||
supported resource manager is not present then ``prte(1)`` relies on a
|
||||
hostfile provided by the user. In the absence of such a hostfile it
|
||||
will run all processes on the localhost.
|
||||
|
||||
If running under a supported resource manager, the ``prte(1)`` process
|
||||
will start the daemon processes (``prted(1)``) on the remote nodes
|
||||
using the corresponding resource manager process starter. If no such
|
||||
starter is available then ``ssh`` (or ``rsh``) is used.
|
||||
|
||||
Minus user direction, PRRTE will automatically map processes in a
|
||||
round-robin fashion by CPU, binding each process to its own CPU. The
|
||||
type of CPU used (core vs hwthread) is determined by (in priority
|
||||
order):
|
||||
|
||||
* user directive on the command line via the HWTCPUS qualifier to
|
||||
the ``--map-by`` directive
|
||||
|
||||
* setting the ``rmaps_default_mapping_policy`` MCA parameter to
|
||||
include the ``HWTCPUS`` qualifier. This parameter sets the default
|
||||
value for a PRRTE DVM |mdash| qualifiers are carried across to DVM
|
||||
jobs started via ``prun`` unless overridden by the user's command
|
||||
line
|
||||
|
||||
* defaulting to ``CORE`` in topologies where core CPUs are defined,
|
||||
and to ``hwthreads`` otherwise.
|
||||
|
||||
By default, the ranks are assigned in accordance with the mapping
|
||||
directive |mdash| e.g., jobs that are mapped by-node will have the
|
||||
process ranks assigned round-robin on a per-node basis.
|
||||
|
||||
PRRTE automatically binds processes unless directed not to do so by
|
||||
the user. Minus direction, PRRTE will bind individual processes to
|
||||
their own CPU within the object to which they were mapped. Should a
|
||||
node become oversubscribed during the mapping process, and if
|
||||
oversubscription is allowed, all subsequent processes assigned to that
|
||||
node will *not* be bound.
|
||||
|
||||
.. include:: /prrte-rst-content/definitions-slots.rst
|
||||
|
||||
.. include:: /prrte-rst-content/definitions-pes.rst
|
||||
Reference in New Issue
Block a user