Cluster with Witness Node

Witness node is a lightweight node only runs etcd which is not schedulable and also not for workloads. The main use case is to form a quorum with the other 2 nodes.

Kubernetes need at least 3 etcd nodes to form a quorum, so Harvester also suggests using at least 3 nodes with similar hardware spec. This witness node feature aims for the edge case that user only have 2 powerful + 1 lightweight nodes thus helping benefit both cost and high availability.

Related Issues

1. #3266 Support 2 Node / Witness Configuration
2. #4840 [ENHANCEMENT] various witness mode related enhancement

References

• HLD: https://github.com/harvester/harvester/blob/master/enhancements/20231113-witness-node-support.md
• rancher-vcluster addon: https://docs.harvesterhci.io/v1.3/advanced/addons/rancher-vcluster

Verification Steps

1. Setup a 2 nodes cluster with default installation role
2. Check nodes’ role
   • node-0: Management node
   • node-1: Compute node (quorum yet)
3. Setup and join 3rd node with witness role
4. Check nodes’ role
   • node-0: Management node
   • node-1: Management node (Reach quorum, promoted)
   • node-2: Witness node
5. Create a VM using default storage class harvester-longhorn with some test data
   • Volume should be Ready but Degraded since replica count is NOT satisfied (expect 3 but only 2 management nodes)
   • Can NOT migrate VM to witness node
   • Can take snapshot
   • Can take backup
6. Enable rancher-vcluster
   • Join Harvester to Rancher
   • Create Nginx deployment
   • Create load balancer for deployment

Expected Results

1. Restart VM, VM and its data should still fine.
2. Restore snapshot, VM and its data should still fine.
3. Restore backup, VM and its data should still fine.
4. rancher-vcluster works.

Notes

1. Except manual setup, another approach is harvester setup configuration https://docs.harvesterhci.io/v1.3/install/harvester-configuration#installrole