We need a STONITH/fencing agent for our Pacemaker cluster. What's a reliable, hardware-agnostic one?

2. fence_mpath (SCSI-3 Persistent Reservations)

• How it works: This is an I/O fencing agent rather than a power-cycling agent. It utilizes SCSI-3 Persistent Reservations (PR) with the "preempt-and-abort" command on shared multipath storage devices. Every node registers a unique key on the shared LUNs. When a node needs to be fenced, the surviving node takes over the reservation and cuts off the misbehaving node's I/O access to the shared disk.
• Hardware Agnostic? Yes. It abstracts away the hardware completely, depending entirely on whether your shared storage array supports standard SPC-3 (or greater) SCSI-3 Persistent Reservations.
• Pros: Perfect for environments where power control (IPMI/PDU) is completely unavailable or restricted, and you already have shared SAN/multipath storage. Prevents split-brain data corruption by locking out the bad node's disk access.
• Cons: Requires storage arrays that properly implement SCSI-3 PR. It cuts off data access rather than power-cycling the whole machine, meaning the fenced node must still be manually or automatically rebooted later if it's merely isolated rather than dead.ubuntu.com

Recommendation Summary

• Use SBD if you can provision a shared block device (or disk) and configure a reliable watchdog timer on your nodes. It is widely considered the most bulletproof, low-latency fencing mechanism for modern clusters.docs.redhat.com
• Use fence_mpath if your infrastructure relies heavily on shared SAN/multipath disks and you cannot use power-based or storage-heartbeat methods, provided your storage backend fully supports SCSI-3 PR.ubuntu.com

To help narrow down the best implementation path, let me know:

• Are you running on bare-metal or in a virtualized/cloud environment?
• Do you have shared storage (like iSCSI, Fibre Channel, or a SAN) available across the cluster nodes?

One important caveat: don't use an agent that SSHes into the target and asks its OS to shut itself down. Pacemaker's documentation specifically warns that a fencing mechanism must not depend on the target node functioning correctly.ClusterLabs

If you have IPMI/Redfish/BMC access, those are also good options, but they're inherently tied to the server's out-of-band management implementation. For genuinely heterogeneous bare-metal hardware, SBD + hardware watchdog is the option I'd investigate first.

If you tell me whether this is 2-node or 3+ node, and whether you have shared storage and/or hardware watchdogs, I can give you a concrete pcs configuration and the failure scenarios it covers.

Key Alternatives depending on your setup:

• fence_ipmilan / fence_ipmi:
  • When to use: If your cluster runs on bare-metal servers from different vendors, but every single server has a standard IPMI/BMC network interface available. It's technically vendor-specific per command, but universally supported across enterprise server hardware.docs.redhat.com+2
• fence_kdump:
  • When to use: As a helper/first-line agent in a fencing topology . It allows a crashing/kernel-panicking node to notify the cluster before getting hard-fenced by a physical/power device, preventing unnecessary hard reboots during a trace dump.documentation.suse.com+1

If you'd like to proceed, let me know:

• Do you have shared storage (SAN, iSCSI, shared disk) available for all cluster nodes?
• Are you running on bare-metal or virtualized infrastructure (e.g., KVM, VMware, AWS)?

I can help you outline the exact configuration steps for SBD or recommend an alternative fit for your environment.

The important caveat

SBD isn't literally hardware-independent: the fencing action ultimately depends on a functioning watchdog on each node. Modern servers commonly expose one such as iTCO_wdt, ipmi_wdt, or vendor-specific watchdogs, but you should verify yours.SUSE Documentation

If your requirement is “works across arbitrary physical servers, VMs, and cloud instances”, I'd instead use the appropriate external fencing agent for each environment (IPMI/Redfish/cloud API), potentially with SBD as a second fencing level. Pacemaker supports multiple fencing mechanisms and fencing levels.Red Hat Documentation+1

So, in shorthand:

Bare-metal heterogeneous cluster → SBD + hardware watchdog VM/cloud/mixed infrastructure → environment-specific power fencing, optionally SBD as fallback Two-node cluster → pay particular attention to quorum/split-brain; diskless SBD alone has limitations.SUSE Documentation

If you tell me whether your nodes are bare metal, VMs, or mixed, and whether you have shared storage, I can give you a concrete pcs stonith configuration.

How to Implement SBD (Watchdog-Only Mode)

If you aren't using shared block devices (SAN/disk) for SBD, you can configure SBD to rely purely on the local watchdog.projects.clusterlabs.org

Note: Ensure your kernel watchdog module (e.g., softdog, iTCO_wdt ) is loaded on all nodes before starting.

1. Install the SBD package:
   On RHEL/CentOS/Fedora: dnf install sbd
   On Debian/Ubuntu: apt install sbd
2. Configure the SBD timeout defaults in /etc/sysconfig/sbd (or distribution equivalent), setting an appropriate SBD_WATCHDOG_TIMEOUT (e.g., 30 seconds).
3. Enable SBD configuration via your cluster management tool (e.g., pcs on RHEL):bash``` pcs stonith sbd enable SBD_WATCHDOG_TIMEOUT=30

Use code with caution.docs.redhat.com+2

If you'd like, let me know:

• What operating system/distribution you are running
• Whether you have shared storage available or prefer pure local watchdog configuration

I can provide the exact command syntax and configuration file templates for your environment.

Choosing the Right Approach

1. If you have out-of-band management network: Use fence_ipmilan . It is native, widely tested, and works across mixed-vendor whitebox or enterprise server hardware.documentation.suse.com
2. If you already have shared storage (and no reliable IPMI/BMC): Use sbd . It avoids dependency on network power switches or flaky BMC firmware, making it exceptionally robust in enterprise database clusters.community.sap.com
3. If you are in a mixed environment with smart rack power: Use fence_snmp or specific PDU agents (fence_apc , etc.) mapped to individual outlet ports.wiki.lustre.org+1

To help narrow down the best fit, tell me:

• Are these bare-metal servers, virtual machines , or cloud-hosted instances?
• Do you have shared storage (like iSCSI/SAN) or a dedicated out-of-band management network available?

How SBD Works in Practice

1. Heartbeat & Timeout: Every cluster node constantly writes to a designated sector on a shared block device (or communicates via SBD slots).
2. The Deadman Switch: If a node gets overloaded, loses network connectivity, or freezes, it fails to update its heartbeat.
3. Self-Reset / Suicide: The peer nodes notice the missed heartbeat and mark the node poisoned. Simultaneously, the local watchdog on the frozen/isolated node triggers a hard reset because the local SBD daemon stopped feeding the watchdog.

If you can share your infrastructure type (e.g., bare-metal with SAN, KVM virtual machines, or public cloud like AWS/Azure), I can provide the exact SBD configuration parameters or recommend whether a cloud-specific fencing agent (fence_aws, fence_azure_arm ) would fit better.

Typical options:

• Best general-purpose physical cluster choice: fence_sbd + shared SAN/iSCSI block device.GitHub
• If you lack shared storage: watchdog-only SBD can work, but requires careful quorum design; production setups should use a real hardware watchdog.ClusterLabs
• Avoid: SSH/shutdown-based fencing agents; they depend on the failed node still functioning and are unsafe for production fencing.clusterlabs.org

Example Pacemaker resource:

For heterogeneous physical servers, SBD is usually the closest thing to a hardware-independent STONITH mechanism.hawk-guide.readthedocs.io