Setting Up High Availability (CARP) in OPNsense

This guide describes how to build a two-node OPNsense high availability cluster using CARP (Common Address Redundancy Protocol). It covers address planning, pfsync, configuration synchronization, DHCP failover with Kea, and the hypervisor and switch settings that cause most CARP failures, particularly when the firewalls are virtual machines.

How OPNsense HA works

An OPNsense HA cluster consists of three separate mechanisms:

  1. CARP. Two nodes share virtual IPs (VIPs). Both nodes are configured with each VIP, but only the current MASTER answers for it. Nodes elect the master by exchanging multicast advertisements. If the master stops advertising, the backup takes over in about a second.
  2. pfsync. The master streams its firewall state table to the backup over a dedicated link, so established connections survive a failover instead of being reset.
  3. XMLRPC config sync. The primary pushes configuration (firewall rules, NAT, VIPs, aliases, service configs) to the secondary, so administration happens on one box.

Clients and DNS never point at either node's real IP. They point at the CARP VIPs, which move to whichever node is currently master.

Address planning

Plan the addressing before touching the GUI. Every interface that participates in HA needs three addresses: a real IP for each node, plus the shared CARP VIP that clients actually use.

WAN

You need a routed block with at least three usable addresses; each additional public service bind requires another. Example with a /29:

No ISP-side change is needed as long as the VIPs are inside a block already routed to your segment.

Internal interfaces

The same pattern applies on every LAN/VLAN: the address clients use as their default gateway becomes a CARP VIP, and each node gets a unique real IP beside it.

Notes:

Hypervisor and switch prerequisites

If your OPNsense nodes are VMs, do this section first. These settings account for most "CARP doesn't work" and "CARP flaps constantly" reports, and none of them are visible from inside OPNsense.

CARP has two requirements that default hypervisor security settings block. The master answers for a virtual MAC address (00:00:5e:00:01:XX, where XX is the VHID) that does not belong to the VM's NIC, and the nodes communicate via multicast (224.0.0.18).

VMware ESXi

On the vSwitch/portgroups carrying CARP interfaces, set all three security options to Accept:

Without these, VIPs show "MASTER" locally but traffic to them is silently dropped by the vSwitch, or the nodes never hear each other's advertisements.

ESXi with two or more physical uplinks: constant flapping

If the vSwitch has two uplinks and promiscuous mode is on, the following loop forms:

  1. A node sends a CARP advertisement out uplink 1.
  2. The physical switch floods the multicast back in on uplink 2.
  3. Promiscuous mode delivers the reflected copy back to the same node that sent it.
  4. The node interprets this as another master and logs MASTER -> BACKUP (more frequent advertisement received), then shortly after BACKUP -> MASTER (master timed out). Every VHID flaps at the same instant, and on IPv6 this can escalate into a duplicate-address-detection storm.

The fix is an ESXi advanced setting that drops multicast arriving on the uplink it did not leave from:

# check (default is 0)
esxcli system settings advanced list -o /Net/ReversePathFwdCheckPromisc

# enable
esxcli system settings advanced set -o /Net/ReversePathFwdCheckPromisc -i 1

Apply it on both hosts, then activate it by toggling Promiscuous Mode Reject → Accept on the affected portgroups. The setting only takes effect when promiscuous mode is re-applied; no reboot is needed, and it persists across reboots.

An alternative, if that setting is not an option: configure the CARP portgroups' NIC teaming with a single active uplink (others standby). That also stops the reflection, at the cost of uplink load balancing.

Proxmox VE

Linux bridges do not filter the way vSwitches do, but Proxmox has its own issues:

Physical switches

Step 1: pfsync link

Give each node an interface on the dedicated sync segment (e.g. a PFSYNC VLAN, 10.10.10.1/30 and 10.10.10.2/30). Confirm the nodes can ping each other across it before going further.

Add the firewall rule. A newly assigned OPNsense interface default-denies everything inbound. Add a pass rule on the PFSYNC interface (source = the pfsync subnet 10.10.10.0/30, protocol any) on both nodes. Without it, state sync fails, along with anything else that runs over this link, such as Kea HA.

Step 2: HA settings (System → High Availability → Settings)

On the primary:

On the secondary:

Two requirements that are easy to miss:

Step 3: CARP VIPs (Firewall → Virtual IPs)

Create all VIPs on the primary; they sync to the secondary. For each:

If you are converting from a single node that used IP alias VIPs for extra addresses, delete those aliases and re-create them as CARP VIPs. Do not leave duplicates behind.

After any VIP rework: run System → High Availability → Status → Synchronize, verify the secondary's VIP list is identical (no leftover /32s or alias duplicates), and reboot both nodes if the VIP set changed significantly.

Healthy state (Interfaces → Virtual IPs → Status): the primary shows every VIP MASTER at freq 1/0; the secondary shows every VIP BACKUP at freq 1/100; nothing flaps.

Step 4: per-node settings

Config sync deliberately does not touch the following. Set these by hand on each box:

If you are converting an existing node, this is where its old interface IPs (now claimed by the VIPs) get replaced with the new node reals. Do this with console access available, since you are editing the addresses you are connected through.

Step 5: point services at the VIPs

ACME / Let's Encrypt in HA

Step 6: DHCP failover with Kea

OPNsense's Kea DHCP supports HA in hot-standby mode: the primary answers all DHCP, both daemons replicate leases to each other over the pfsync link, and the standby takes over with the full lease table intact.

Configure identical HA peer entries on both nodes (they sync from the primary):

Key points:

Verify: both Kea daemons are running with no bind errors, the secondary's Leases page shows the same leases as the primary, and the HA state reads hot-standby with the partner in normal/ready.

Step 7: test the failover

  1. Prerequisite: confirm every synced-config plugin (HAProxy etc.) is installed and running on the secondary. Config sync will push a full HAProxy config to a box with no HAProxy daemon; the VIPs will fail over and nothing will answer on them.
  2. Baseline: primary all MASTER, secondary all BACKUP, services up.
  3. On the primary: Enter Persistent CARP Maintenance Mode (Interfaces → Virtual IPs → Status), or reboot it.
  4. The secondary should take MASTER within a couple of seconds. Browse your hosted services, make an outbound connection, and renew a DHCP lease. Everything should keep working, and pfsync should keep existing connections alive.
  5. Exit maintenance mode. The primary reclaims MASTER (the advskew difference makes it the preferred master).

Repeat this test after any CARP or VIP change.

Troubleshooting quick reference

Known limitations

Advertisement