Support private registry for Rancher agent image in Air-gap

• Related issues: #2176 [Enhancement] Air-gap operation: Support using a private registry for Rancher agent image

Category:

• Rancher Integration

Verification Steps

Environment Setup

1. Use vagrant-pxe-harvester to create a harvester cluster.
2. Create another VM myregistry and set it in the same virtual network.
3. In myregistry VM:
   • Install docker.
   • Run following commands:

   mkdir auth
   docker run \
   --entrypoint htpasswd \
   httpd:2 -Bbn testuser testpassword > auth/htpasswd
   
   mkdir -p certs
   
   openssl req \
   -newkey rsa:4096 -nodes -sha256 -keyout certs/domain.key \
   -addext "subjectAltName = DNS:myregistry.local" \
   -x509 -days 365 -out certs/domain.crt
   
   sudo mkdir -p /etc/docker/certs.d/myregistry.local:5000
   sudo cp certs/domain.crt /etc/docker/certs.d/myregistry.local:5000/domain.crt
   
   docker run -d \
   -p 5000:5000 \
   --restart=always \
   --name registry \
   -v "$(pwd)"/certs:/certs \
   -v "$(pwd)"/registry:/var/lib/registry \
   -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
   -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
   -v "$(pwd)"/auth:/auth \
   -e "REGISTRY_AUTH=htpasswd" \
   -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
   -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
   registry:2
   

4. Set IP and domain in /etc/hosts to all VM (harvester and myregistry). Remember to change the IP.

   # vim /etc/hosts
   192.168.0.50 myregistry.local
   

5. Login, pull, and push nginx image in myregsitry VM:

   # username: testuser, password: testpassword
   docker login myregistry.local:5000
   docker pull nginx:latest
   docker tag nginx:latest myregistry.local:5000:/nginx:latest
   docker push myregistry.local:5000/nginx:latest
   docker pull nginx:1.22
   docker tag nginx:latest myregistry.local:5000:/nginx:1.22
   docker push myregistry.local:5000/nginx:1.22
   

6. Copy certs/domain.crt content in myregistry VM and paste it to additional-ca setting.

Test Plan 1

1. Update the Harvester containerd-registry setting to use private registry

   {
   "Mirrors": {
       "docker.io": {
       "Endpoints": [
           "https://myregistry.local:5000"
       ],
       "Rewrites": null
       }
   },
   "Configs": {
       "myregistry.local:5000": {
       "Auth": null,
       "TLS": {
           "CAFile": "",
           "CertFile": "",
           "KeyFile": "",
           "InsecureSkipVerify": false
       }
       }
   },
   "Auths": null
   }
   

2. Open K9s -> search -> secrets -> containerd -> y

3. Check content in cattle-system/harvester-containerd-registry secret is changed.

4. Search jobs -> containerd -> l -> tail

5. Check there are new jobs to auto apply the new containerd-registry setting.

6. Apply following yaml in file apiVersion: apps/v1 kind: Deployment metadata: name: myregistry-nginx spec: selector: matchLabels: image: myregistry-nginx template: metadata: labels: image: myregistry-nginx spec: containers: - imagePullPolicy: Always image: myregistry.local:5000/nginx:latest name: nginx ``` harvester-node-0:~ # kubectl apply -f deploy.yaml deployment.apps/myregistry-nginx created

    ```
   

7. search deployments -> default namespace -> myregsistry-nginx -> describe -> shift + g

8. Check the nginx can be deployed.

Test Plan 2

1. Click the default value to update the Harvester containerd-registry setting to default empty
2. In K9s
3. Check content in cattle-system/harvester-containerd-registry secret is changed.
4. Check there are new jobs to auto apply the new containerd-registry setting.
5. Search secrets -> containerd -> e
6. Clean registry.yaml data field

   # Please edit the object below. Lines beginning with a '#' will be ignored,
   # and an empty file will abort the edit. If an error occurs while saving this file will be
   # reopened with the relevant failures.
   #
   apiVersion: v1
   data:
       registries.yaml: ""
   kind: Secret
   metadata:
       creationTimestamp: "2022-10-04T08:51:47Z"
       name: harvester-containerd-registry
       namespace: cattle-system
       resourceVersion: "42088"
       uid: 8445547f-430a-474e-9783-dc6b16914eb7
   type: Opaque
   

7. Search -> deployments -> find nginx-1.2.2
8. Check cannot pull nginx-1.2.2 image

Test Plan 3

1. Add back the Harvester containerd-registry setting to use private registry again
2. Check content in cattle-system/harvester-containerd-registry secret is changed.
3. Check there are new jobs to auto apply the new containerd-registry setting.
4. Check nginx 1.22 deployment is automatically back to running.

Expected Results

1. Update the Harvester containerd-registry setting with private registry, confirm can pull image and deploy nginx service correctly from private registry

   

2. Change the Harvester containerd-registry to default value, the nginx 1.22 cannot be deployed since no private registry assigned.

   

3. Add back the the Harvester containerd-registry setting to private registry, confirm can pull image and deploy nginx service correctly from private registry