collectives

The collectives module exposes a set of distributed collective operations. Those can be used to exchange data between participating sites in a coordinated way. At this point the module exposes the following collective primitives:

• hpx::collectives::all_gather: receives a set of values from all participating sites.

• hpx::collectives::all_reduce: performs a reduction on data from each participating site to each participating site.

• hpx::collectives::all_to_all: each participating site provides its element of the data to collect while all participating sites receive the data from every other site.

• hpx::collectives::broadcast_to and hpx::collectives::broadcast_from: performs a broadcast operation from a root site to all participating sites.

• hpx::collectives::exclusive_scan: performs an exclusive scan operation on a set of values received from all call sites operating on the given base name.

• hpx::collectives::gather_here and hpx::collectives::gather_there: gathers values from all participating sites.

• hpx::collectives::inclusive_scan: performs an inclusive scan operation on a set of values received from all call sites operating on the given base name.

• hpx::collectives::reduce_here and hpx::collectives::reduce_there: performs a reduction on data from each participating site to a root site.

• hpx::collectives::scatter_to and hpx::collectives::scatter_from: receives an element of a set of values operating on the given base name.

• hpx::lcos::broadcast: performs a given action on all given global identifiers.

• hpx::distributed::barrier: distributed barrier.

• hpx::lcos::fold: performs a fold with a given action on all given global identifiers.

• hpx::distributed::latch: distributed latch.

• hpx::lcos::reduce: performs a reduction on data from each given global identifiers.

• hpx::lcos::spmd_block: performs the same operation on a local image while providing handles to the other images.

See the API reference of the module for more details.

Hierarchical collectives

Hierarchical communicators split a collective into sub-communicators arranged as a tree. This reduces the number of participants in each individual operation, but it also means that one user-visible hierarchical call can touch several internal communicators.

Generation model

Hierarchical collectives use two internal generations for each user generation. For a user generation k, the first internal phase uses 2k - 1 and the second uses 2k. Single-pass hierarchical operations and the inter-group exchange of hpx::collectives::all_to_all advance the communicator by two generations in one step so that shared hierarchical communicators stay aligned with two-phase collectives.

When a hierarchical communicator instance is shared between collective operations, callers must pass explicit, strictly consecutive positive generation numbers. Two-phase hierarchical collectives, including hpx::collectives::all_gather, hpx::collectives::all_reduce, hpx::collectives::all_to_all, hpx::collectives::inclusive_scan, hpx::collectives::exclusive_scan, and hpx::distributed::barrier, reject the default generation because their phase generations are derived from the explicit user generation.