Upgrade support of audit and event log

• Related issues: #2750 [FEATURE] Upgrade support of audit and event log

Category:

• Logging Audit

Verification Steps

1. Prepare v1.0.3 cluster, single-node and multi-node need to be tested separately
2. Upgrade to v1.1.0-rc2 / master-head
3. The upgrade should be successful, if not, check log and POD errors
4. After upgrade, check following PODs and files, there should be no error

Expected Results

Check both Single and Multi nodes upgrade of the following:

1. Check the following files and pods have no error

2. logging related pods

   n1-103:~ # kubectl get pods -n cattle-logging-system
   NAME                                             READY   STATUS    RESTARTS      AGE
   harvester-default-event-tailer-0                 1/1     Running   0             47m
   rancher-logging-574448c578-wv9wd                 1/1     Running   0             47m
   rancher-logging-kube-audit-fluentbit-dnxzr       1/1     Running   2 (38m ago)   59m
   rancher-logging-kube-audit-fluentbit-s9bfj       1/1     Running   2 (29m ago)   59m
   rancher-logging-kube-audit-fluentbit-snrml       1/1     Running   2 (49m ago)   59m
   rancher-logging-kube-audit-fluentd-0             2/2     Running   0             36m
   rancher-logging-rke2-journald-aggregator-c4kk4   1/1     Running   2 (49m ago)   59m
   rancher-logging-rke2-journald-aggregator-rc4gf   1/1     Running   2 (29m ago)   59m
   rancher-logging-rke2-journald-aggregator-sc56f   1/1     Running   2 (38m ago)   59m
   rancher-logging-root-fluentbit-fh9v9             1/1     Running   2 (49m ago)   59m
   rancher-logging-root-fluentbit-v65sz             1/1     Running   2 (38m ago)   59m
   rancher-logging-root-fluentbit-xkp8p             1/1     Running   2 (29m ago)   59m
   rancher-logging-root-fluentd-0                   2/2     Running   0             36m
   

3. kube-audit files

4. Check the kube-apiserver has param of audit

       n1-103:~ # ps aux | grep kube-apiserver
       root      2819 41.7  2.6 3468424 2196848 ?     Ssl  12:25  56:05 kube-apiserver --audit-policy-file=/etc/rancher/rke2/config.yaml.d/92-harvester-kube-audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log
   

5. The /var/lib/rancher/rke2/server/logs/audit.log have increasing content

   n1-103:~ # du -sh /var/lib/rancher/rke2/server/logs/audit.log
   8.0M	/var/lib/rancher/rke2/server/logs/audit.log
   
   n1-103:~ # du -sh /var/lib/rancher/rke2/server/logs/audit.log
   8.5M	/var/lib/rancher/rke2/server/logs/audit.log
   

6. config file exists

   n1-103:~ # cat /etc/rancher/rke2/config.yaml.d/90-harvester-server.yaml
   cni: multus,canal
   cluster-cidr: 10.52.0.0/16
   service-cidr: 10.53.0.0/16
   cluster-dns: 10.53.0.10
   tls-san:
       - 192.168.50.199
   audit-policy-file: /etc/rancher/rke2/config.yaml.d/92-harvester-kube-audit-policy.yaml
   

   apiVersion: audit.k8s.io/v1
   kind: Policy
   omitStages:
       - "ResponseStarted"
       - "ResponseComplete"
   rules:
       # Any include/exclude rules are added here
       # A catch-all rule to log all other (create/delete/patch) requests at the Metadata level
       - level: Metadata
       verbs: ["create", "delete", "patch"]
       omitStages:
           - "ResponseStarted"
           - "ResponseComplete"
   

7. /oem/99_custom.yaml is patched

   n1-103:~ # grep "92-harvester" /oem/99_custom.yaml -5
               cluster-cidr: 10.52.0.0/16
               service-cidr: 10.53.0.0/16
               cluster-dns: 10.53.0.10
               tls-san:
                   - 192.168.50.199
               audit-policy-file: /etc/rancher/rke2/config.yaml.d/92-harvester-kube-audit-policy.yaml
               encoding: ""
               ownerstring: ""
           - path: /etc/rancher/rke2/config.yaml.d/90-harvester-agent.yaml
               permissions: 384
               owner: 0
   --
                   valuesContent: |-
                   flannel:
                       iface: harvester-mgmt
               encoding: ""
               ownerstring: ""
           - path: /etc/rancher/rke2/config.yaml.d/92-harvester-kube-audit-policy.yaml
               permissions: 384
               owner: 0
               group: 0
               content: |
               apiVersion: audit.k8s.io/v1
   

   - path: /etc/rancher/rke2/config.yaml.d/92-harvester-kube-audit-policy.yaml
               permissions: 384
               owner: 0
               group: 0
               content: |
               apiVersion: audit.k8s.io/v1
               kind: Policy
               omitStages:
                   - "ResponseStarted"
                   - "ResponseComplete"
               rules:
                   # Any include/exclude rules are added here
                   # A catch-all rule to log all other (create/delete/patch) requests at the Metadata level
                   - level: Metadata
                   verbs: ["create", "delete", "patch"]
                   omitStages:
                       - "ResponseStarted"
                       - "ResponseComplete"
               encoding: ""
               ownerstring: ""
   

8. Check the UI of Monitoring & Logging work as in fresh installed cluster after upgrade to v1.1.0-rc2

9. Cluster Outputs

10. Cluster Flows

11. Logging configuration

12. Outputs

13. Flow

14. Monitoring configuration - Altermanager

15. Altermanager config