hpx/supervision_dispatch/server/registry.hpp#

Defined in header hpx/supervision_dispatch/server/registry.hpp.

See Public API for a list of names and headers that are part of the public HPX API.

namespace hpx
namespace supervision#

Fenced action dispatch for the supervision subsystem.

This header provides the machinery to dispatch an HPX action to a target component while enforcing supervision “fencing” semantics: once a target has latched a terminal event for a given epoch, no further actions for that epoch may execute against it. Dispatch performs a cheap, non-authoritative admission check on the caller’s locality and an authoritative re-check on the target’s own locality (via hpx::colocated), immediately before invoking the wrapped action, closing the race between admission and invocation.

Types and functions implementing HPX’s supervision and fenced dispatch facilities.

Functions

inline bool operator==(joined_peer const &lhs, joined_peer const &rhs) noexcept#
inline bool operator!=(joined_peer const &lhs, joined_peer const &rhs) noexcept#
std::ostream &operator<<(std::ostream &strm, joined_peer const &peer)#
struct joined_peer#
#include <registry.hpp>

Pairs the two values a caller obtains from a successful registry::join() and needs to perform a fenced dispatch.

See also

dispatch_work()

Public Members

hpx::id_type target#

The real, colocatable destination for the wrapped action.

The peer’s locality id (i.e. peer_locality, the value originally passed into registry::join()). Forwarded to hpx::colocated() and used as the destination of hpx::sync(act, target, ts...) inside dispatch_work().

std::uint64_t join_epoch = 0#

The epoch at which this peer was joined.

Recorded by registry::join() and used to identify which registry entry a later registry::leave() call refers to - so that a racing re-join of the same peer locality (which mints a fresh join_epoch of its own) cannot be mistaken for the join this joined_peer was returned from.

namespace server#
struct peer_snapshot#
#include <registry.hpp>

Plain-data view of a single joined peer, safe to hand out to callers outside the registry (unlike peer_entry, does not expose the ready/evict_pending bookkeeping used internally to coordinate concurrent join()/evict_peer() calls).

Public Members

hpx::id_type peer_locality#

The peer locality this entry was joined against.

std::uint64_t join_epoch#

The epoch at which peer_locality was joined.

std::uint64_t generation#

Identifies this particular registry entry. A peer that is left and joined again gets a new one, even if its join_epoch is the same. See registry::evict_fenced().

class registry : public hpx::components::component_base<registry>#

Public Functions

registry()#
joined_peer join(hpx::id_type const &peer_locality)#

Joins a peer locality and creates or reuses its local supervision state.

Parameters:

peer_locality – The peer locality to observe.

std::vector<peer_snapshot> snapshot_peers() const#

Returns a point-in-time snapshot of all fully joined, non-evicting peers.

Intended for a future failure-detection poller (or other local consumers) that needs a stable, lock-safe list of peers to iterate over without reaching into peers_ directly. Entries mid-join (ready == false) or already scheduled for eviction (evict_pending == true) are excluded, so a poller never observes a half-registered or torn-down peer.

Returns:

A snapshot of every peer entry with ready == true and evict_pending == false at the time of the call.

void leave(hpx::id_type const &peer_locality, std::uint64_t join_epoch)#

Removes a previously joined peer.

Parameters:
  • peer_locality – The peer locality to leave.

  • join_epoch – Must match the join epoch currently recorded for peer_locality (i.e. the epoch returned by the corresponding join()). If peer_locality has since been re-joined (fresh epoch) or already evicted, this call is a no-op.

void evict_fenced(hpx::id_type const &peer_locality, std::uint64_t generation)#

Removes a joined peer whose local supervision state has latched a terminal event without the peer itself reporting one, i.e. one that failure detection fenced. The terminal state is left in place as a tombstone (see cleanup_peer()), so the peer stays fenced.

Ordering with the peer’s lifecycle notifications: the terminal check and the removal are validated against the entry’s lifecycle mirroring (see peer_entry::mirror_sequence), so exactly one of the following holds for every notification the entry’s lifecycle observer receives:

  • its mirrored state is visible to the terminal check, so a peer that came back with a newer, non-terminal epoch is not evicted;

  • it was in flight or finished between the terminal check and the removal, in which case this call backs off without evicting anything (the next failure-detection sweep calls it again if the state is still terminal by then);

  • it arrived after the removal, in which case it is not mirrored at all, so the tombstone stays in place and the peer has to join again to be monitored, as after any other eviction.

The local state can also change before the lifecycle observer runs: a registry that joined its own locality shares that state with the peer’s own lifecycle events, and several registries on one locality joining the same peer share it with each other’s mirroring. To cover that, the local state is queried a second time after the terminal check, and the entry is removed only if it is still the state the terminal check saw (same epoch, event sequence number, and event). So for every change of the local state whose observer has not announced a mirroring by the time the entry is removed:

  • if it happened before the second query, that query sees it, and this call backs off, leaving the entry and its observers in place;

  • if it happened after the second query, it overlaps this call and is treated as if it happened after the removal: its notification is not mirrored, and the peer has to join again to be monitored. For a registry that joined its own locality, the local state then reflects the peer’s newer lifecycle instead of the tombstone, exactly as if the peer had started that lifecycle right after this call returned.

No lock is held while the local supervision state is queried or while the observers are unregistered on the peer’s locality.

Parameters:
  • peer_locality – The peer locality to remove.

  • generation – The peer_snapshot::generation of the entry to remove. If peer_locality has since been evicted, or left and joined again (which creates a new entry, even if its join epoch is the same), or its local supervision state is no longer terminal, this call is a no-op.

Protected Functions

std::tuple<hpx::id_type, std::uint64_t, bool> reserve_ownership(hpx::id_type const &peer_locality)#
std::pair<hpx::id_type, hpx::id_type> register_observers(hpx::id_type const &peer_locality, std::uint64_t generation)#
void evict_peer(hpx::id_type const &peer_locality, std::uint64_t join_epoch, std::uint64_t generation, hpx::id_type keep_alive, bool preserve_terminal_state)#

Protected Static Functions

static void cleanup_peer(hpx::id_type const &peer_locality, hpx::id_type const &lifecycle_observer, hpx::id_type const &activity_observer, bool preserve_terminal_state)#

Unregisters a peer’s observers and, unless preserve_terminal_state is set, removes its locally tracked supervision state.

Performs the actual peer teardown deferred by evict_peer(): unregisters lifecycle_observer and activity_observer on peer_locality, and removes peer_locality's locally tracked supervision state unless preserve_terminal_state is true, in which case that state - which was last set to the peer’s terminal event by register_observers()’s lifecycle callback - is left in place as a tombstone rather than reset to “unknown”. This lets any admission check still in flight against peer_locality (e.g. check_admission()) keep observing the fenced/terminal outcome after eviction has run, instead of racing against it. A subsequent join() re-seeding this locality with a fresh epoch naturally overwrites the tombstone. Kept separate from evict_peer() because evict_peer() only posts this work as a task (via hpx::post()) rather than performing it inline.

Parameters:
  • peer_locality – The locality that owns the observers, and whose local supervision state is removed.

  • lifecycle_observer – The lifecycle observer id to unregister.

  • activity_observer – The activity observer id to unregister.

  • preserve_terminal_state – If true, leave peer_locality’s local supervision state in place instead of removing it.

Protected Static Attributes

static constexpr std::uint64_t any_generation = 0#

Private Members

mutable hpx::spinlock mtx_#
hpx::lcos::local::detail::condition_variable cond_#
std::map<hpx::id_type, peer_entry> peers_#
std::uint64_t next_generation_ = 0#
struct peer_entry#

Internal bookkeeping for a single joined (or joining) peer.

Not exposed outside the registry; see peer_snapshot for the plain-data subset safe to hand to external callers.

Public Functions

inline explicit peer_entry(hpx::id_type const &peer_locality)#

Public Members

hpx::id_type const peer_locality#

The locality that owns this peer’s. Surfaced read-only via peer_snapshot::peer_locality.

hpx::id_type lifecycle_observer#

The observer id registered on peer_locality to receive this peer’s terminal lifecycle notification (drives evict_peer() via hpx::post()).

hpx::id_type activity_observer#

The observer id registered on peer_locality to receive this peer’s activity notifications (used to keep peer_locality’s local state current between terminal events).

std::uint64_t join_epoch = 0#

The epoch at which this peer was joined, recorded by join() and used by evict_peer() (instead of shadow) to identify and used by evict_peer() to identify which entry a deferred eviction refers to.

bool ready = false#

False while a join() call has reserved ownership of this peer’s locality (i.e. is in the process of performing the remote observer registrations) but has not yet filled in the rest of the entry. Concurrent join() calls for the same peer wait on cond_ instead of racing to register duplicate observers.

bool evict_pending = false#

True once a terminal notification has been observed for this peer but eviction has not yet completed (deferred via hpx::post() to evict_peer()). Ensures a terminal notification racing ahead of join() completion is deferred instead of dropped, and lets snapshot_peers() exclude a peer that is already tearing down.

std::uint64_t generation = 0#

Unique for each entry this registry creates, surfaced via peer_snapshot::generation. Lets evict_fenced() tell the entry it was asked to remove apart from a later one for the same peer locality.

std::size_t mirrors_in_flight = 0#

The number of lifecycle notifications the observer installed by register_observers() is mirroring onto peer_locality’s local supervision state right now. Incremented before the callback touches that state and decremented once it is done, both under mtx_; cond_ is notified when it drops to zero, for evict_peer().

std::uint64_t mirror_sequence = 0#

The number of lifecycle notifications mirrored for this entry so far, incremented under mtx_ once the mirrored state has been published. evict_fenced() reads it before its terminal check and only removes the entry if nothing was in flight then, nothing is in flight at the removal, and the count has not changed in between, so a newer lifecycle transition cannot slip in between the check and the removal unnoticed.

bool evict_preserve_terminal_state = false#

The preserve_terminal_state argument evict_peer() was called with when it set evict_pending above, remembered so that join()’s deferred cleanup_peer() call (once the join in progress completes) preserves or drops peer_locality’s local supervision state consistently with what the racing terminal notification requested.

std::uint64_t evict_pending_epoch = 0#

The join_epoch evict_peer() was called with when it set evict_pending above. join() only carries out the pending eviction if the entry is still at this epoch once the lifecycle notifications being mirrored have finished, so an eviction for an older epoch cannot remove a peer that came back meanwhile.