VM Import/Migration

Related issues: #2274 [Feature] VM Import/Migration

Test Information

Test Environment:

1 node harvester on local kvm machine
Harvester version: v1.1.0-rc1
Vsphere: 7.0
Openstack: Simulated using running devstack
Download kubeconfig for harvester cluster

Environment Setup

Prepare Harvester master node
Prepare vsphere setup (or use existing setup)
Prepare a devstack cluster (Openstack 16.2) (stable/train)

OpenStack Setup

Prepare a baremetal or virtual machine to host the OpenStack service
For automated installation on virtual machine, please refer to the cloud init user data in https://github.com/harvester/tests/issues/522#issuecomment-1654646620
For manual installation, we can also follow the command in the cloud init user data

OpenStack troubleshooting

If you failed create volume with the following error message Error: Failed to perform requested operation on instance "opensuse", the instance has an error status: Please try again later [Error: Build of instance 289d8c95-fd99-42a4-8eab-3a522e891463 aborted: Invalid input received: Invalid image identifier or unable to access requested image. (HTTP 400) (Request-ID: req-248baac7-a2de-4c51-9817-de653a548e3b)].

You can use these steps to configure OpenStack service

Check the certificate file exists

sudo cat /var/snap/microstack/common/etc/ssl/certs/cacert.pem

Run the following command to configure microstack service

1. microstack init --auto --control --setup-loop-based-cinder-lvm-backend
    --loop-device-file-size 100
2. snap restart microstack.cinder-{uwsgi,scheduler,volume}
3. sed -i 's/client_max_body_size 0/client_max_body_size 4G/g' /var/snap/microstack/common/etc/nginx/snap/nginx.conf
4. snap restart microstack.nginx

Check the cinder volume state

$ journalctl -xef -u snap.microstack.cinder-volume shows

$ lvextend -r -l +100%FREE /dev/cinder-volumes/cinder-volumes-pool

Set the ca certificate file

$ sudo tee /var/snap/microstack/common/etc/cinder/cinder.conf.d/glance.conf <<EOF
[DEFAULT]
glance_ca_certificates_file = /var/snap/microstack/common/etc/ssl/certs/cacert.pem
EOF

Restart the cinder service

sudo snap restart microstack.cinder-{uwsgi,scheduler,volume}

sudo cat /var/snap/microstack/common/etc/cinder/cinder.conf.d/glance.conf

Verification Steps

Verify VmwareSource

Define a secret for your vmware cluster

apiVersion: v1
kind: Secret
metadata: 
  name: vsphere-credentials
  namespace: default
stringData:
  "username": "user"
  "password": "password"

Define a VmwareSource Object

apiVersion: migration.harvesterhci.io/v1beta1
kind: VmwareSource
metadata:
  name: vcsim
  namespace: default
spec:
  endpoint: "https://vscim/sdk"
  dc: "DCO"
  credentials:
    name: vsphere-credentials
    namespace: default

The controller will try and list VM’s in the endpoint vmware cluster, and datacenter. If the details of credentials, endpoint and datacenter are correct, then the controller will mark the cluster as ready

$ kubectl get vmwaresource.migration 
NAME      STATUS
vcsim   clusterReady

Verify OpenstackSource

Access the Openstack dashboard
Create some images
Create key pairs
Create an empty volume to ensure the microstack.cinder services works well (if failed, check the OpenStack troubleshooting section)
Launch a vm instance, ensure the vm can be created without error message
Enable the vm-import-controller addon on Harvester
Open Compute -> System -> System Information page, find the Services tab
Check the Region in the page (eg. microstack)
Check the endpoint of the identity item
Access the Harvester node machine

Define a secret for your Openstack cluster in yaml file and use kubectl to create the crd

apiVersion: v1
kind: Secret
metadata: 
  name: devstack-credentials
  namespace: default
stringData:
  "username": "user" #dashboard username
  "password": "password" #dashboard password
  "project_name": "admin"
  "domain_name": "default"

Define an OpenstackSource Object in yaml file and use kubectl to create the crd

apiVersion: migration.harvesterhci.io/v1beta1
kind: OpenstackSource
metadata:
  name: devstack
  namespace: default
spec:
  endpoint: "https://<dashboard ipv4>:5000/v3"  
  region: "microstack" # value found in OpenStack dashboard
  credentials:
    name: devstack-credentials
    namespace: default

The controller will try and list the VM’s in the OpenstackSource project in the specified region, using the information from the credentials.

In this is successful the OpenstackSource will be marked as ready.

$ kubectl get openstacksource.migration
NAME       STATUS
devstack   clusterReady

VirtualMachine Import using VmwareSource

A VM import can be triggered by the creation of a VirtualMachineImport object as follows:

apiVersion: migration.harvesterhci.io/v1beta1
kind: VirtualMachineImport
metadata:
  name: alpine-export-test
  namespace: default
spec: 
  virtualMachineName: "alpine-export-test"
  networkMapping:
  - sourceNetwork: "dvSwitch 1"
    destinationNetwork: "default/vlan1"
  - sourceNetwork: "dvSwitch 2"
    destinationNetwork: "default/vlan2"
  sourceCluster: 
    name: vcsim
    namespace: default
    kind: VmwareSource
    apiVersion: migration.harvesterhci.io/v1beta1

The controller will check that the source is ready, before the virtualmachine import is initiated.

In case the VmwareSource is not ready, then the object will be requeued until the VmwareSource becomes ready

If the VmwareSource is ready, then the VirtualMachine identified by the spec.virtualMachineName will be powered off

The VirtualMachine will be exported as an ovf, the disks will be convered to raw images.

The raw images are exported via http endpoint for the vm-import-controller service and used to define a Harvester VirtualMachineImage. Each disk is imported as a VirtualMachineImage in harvester

Once VirtualMachineImages are “Active” in harvester, a PVC claim will be created for each VirtualMachineImage.

The new PVC Claim is used to create a kubevirt VirtualMachine.

As part of the VirtualMachine creation, the CPU, memory specs from the original VM are copied to create the new VM.

All network interfaces, including their mac addresses are copied into network interfaces to kubevirt VirtualMachine object.

If the VirtualMachine network interfaces source network matches the list of networks available in spec.networkMapping then the new network interface is attached to the destinationNetwork.

If there is no networkMapping defined, or no matching sourceNetwork is found then the network interface in the original VM is attached to the Harvester Management Network.

Once the VM object is created, the VirtualMachine is powered on, and the controller waits for the VirtualMachine to be running

VirtualMachine Import using OpenstackSource

Create the vlan1 network on Harvester and ensure can reach the OpenStack dashboard

A VM import can be triggered by the creation of a VirtualMachineImport object as follows:

apiVersion: migration.harvesterhci.io/v1beta1
kind: VirtualMachineImport
metadata:
  name: openstack-demo
  namespace: default
spec: 
  virtualMachineName: "openstack-demo" #Name or UUID for instance
  networkMapping:
  - sourceNetwork: "shared"
    destinationNetwork: "default/vlan1"
  - sourceNetwork: "public"
    destinationNetwork: "default/vlan2"
  sourceCluster: 
    name: devstack
    namespace: default
    kind: OpenstackSource
    apiVersion: migration.harvesterhci.io/v1beta1

Here is the example:

apiVersion: migration.harvesterhci.io/v1beta1                              
kind: VirtualMachineImport
metadata:
  name: openstack-demo
  namespace: default
spec: 
  virtualMachineName: "opensuse" #Name or UUID for instance
  networkMapping:
  - sourceNetwork: "external"
    destinationNetwork: "default/vlan1"
  sourceCluster: 
    name: microstack
    namespace: default
    kind: OpenstackSource
    apiVersion: migration.harvesterhci.io/v1beta1

After the VirtualMachineImport created, you can find Harvester create a new image from the OpenStack
The OpenStack vm would be imported to Harvester