Node disk manager should prevent too many concurrent disk formatting occur within a short period

Ref: https://github.com/harvester/harvester/issues/1831

Criteria

  • exceed the maximum, there should have requeue devices which equals the exceeds
  • hit the maximum, there should not have requeue devices
  • less than maximum, there should not have requeue devices

image

Verify Steps:

  1. Install Harvester with any node having at least 6 additional disks
  2. Login to console and execute command to update log level to debug and max-concurrent-ops to 1 (On KVM environment, we have to set to 1 to make sure the requeuing will happen.)
    • kubectl patch ds -n harvester-system harvester-node-disk-manager --type=json -p'[{"op":"replace", "path":"/spec/template/spec/containers/0/command", "value": ["node-disk-manager", "--debug", "--max-concurrent-ops", "1"]}]'
  3. Watching log output by executing kubectl get pods -A | grep node-disk | awk '{system("kubectl logs -fn "$1" "$2)}'
  4. Login to dashboard then navigate and edit host to add more than 1 disks
  5. In the console log, should display Hit maximum concurrent count. Requeue device <device id>
  6. In the dashboard, disks should be added successfully.
  7. Login to console and execute command to update log level to debug and max-concurrent-ops to 2
    • kubectl patch ds -n harvester-system harvester-node-disk-manager --type=json -p'[{"op":"replace", "path":"/spec/template/spec/containers/0/command", "value": ["node-disk-manager", "--debug", "--max-concurrent-ops", "2"]}]'
  8. Watching log output by executing kubectl get pods -A | grep node-disk | awk '{system("kubectl logs -fn "$1" "$2)}'
  9. Login to dashboard then navigate and edit host to add 2 disks
  10. In the console log, there should not display Hit maximum concurrent count. Requeue device <device id>
  11. In the dashboard, disks should be added successfully.
  12. Login to console and execute command to update log level to debug
    • kubectl patch ds -n harvester-system harvester-node-disk-manager --type=json -p'[{"op":"replace", "path":"/spec/template/spec/containers/0/command", "value": ["node-disk-manager", "--debug"]}]'
  13. Watching log output by executing kubectl get pods -A | grep node-disk | awk '{system("kubectl logs -fn "$1" "$2)}'
  14. Login to dashboard then navigate and edit host to add less than 5 disks
  15. In the console log, there should not display Hit maximum concurrent count. Requeue device <device id>
  16. In the dashboard, disks should be added successfully.