• Libvirt domains are defined via XML
• Domains defined by Nova are persistent
• XML is re-generated on Hard Reboot
• Manual costomerizations will be overriden
• Logging occurs per-domain
• /var/log/libvirt/qemu/instance_name.log

3. Virsh 

• virsh is the command line tool for Libvirt
• Consumes the same API referenced earlier via C
• virsh list
• List all domains by name
• virsh domname uuid
• Returns the instance_name
• virsh dumpxml uuid
• Configuration for an individual domain

• Translated by Libvirt to ultimately call QEMU with the right arguments
• The XML generated by Nova ends up being modified, fairly heavility, by Libvirt upon definition
• Key difference between “active” and “inactive” XML
• Numerous values are derived at time of instance start
• Output from an active “dumpxml” will likely fail to define
• Use `virsh dumpxml –inactive uuid `

5. Libvirtd Configuration

• Critically important information tuning for Libvirt 1.1.x and newer, example
• max_clients = 50

• Verify the storage backend is appropriate for the migration type
• Perform a shared storage check for normal migrations
• Do the inverse for block migrations
• Checks are run on both the source and destination, orchestrated via RPC calls from the scheduler
• On the destination
• Create the necessary volume connections
• If block migration, create the instance directory, populate missing backing files from Glance and create empty instance disks
• On the source
• Initiate the actual live migration
• Upon complete
• Generate the Libvirt XML and define it on the destination

2. Migrations

• Why migration
• Operations
• Key to performing non-distruptive work
• Re-balancing workloads and resources
• Expectations versus reality
• Special snowflakes
• Ephemeral instance and the “cloud” way
• Type of migration
• Migrate
• Completely “cold”, libvirt does almost nothing
• Share code path with “resize”
• Extremely brittle (users SSh and copies files around)
• Live migration
• Orchestrated almost entirely by Libvirt (via DomainMigrateToURI)
• Block migration
• Similar code path as live migraiton
• More risky and brittle (disks are moving along with state)

3. Live Migrations

• Nova offloads capabilities comparisons to Libvirt
• The API equivalent of virsh capabilities is run by the scheduler on the source and destination; 
• Nova live migraiton
• Important config options
• Live_migration_flat =+ VIR_MIGRATE_LIVE
• block_migration_flag=+ VIR_MIGRATE_LIVE
• Standardized virtual CPU flags
• libvirt_cpu_model = custome
• libvirt_cpu_model = cpu64-rhel6
• “Max Downtime” (not currently tunable)
• Look for upstream patches soon
• Qemu will keep doing when the cut can be done in “30” millseconds

4. Brittle Operations

• Any long running, synchronous tasks
• All migrations (memory sync, disk sync, etc)
• No graceful way to stop services
• Most prone to failure
• Migrate and resize
• Live migraiton (block or otherwise)
• Instance snapshot

• Always investigate before forcing actions
• Look at the log for excpetions
• Check whether an instance is running on multiple hypervisors
• Nova reset-state –active and `nova reboot –hard can go a long way
• Sometime, brute force is going to be required
• Kill -9 qumu or kvm processes
• After the database records, commonly `host`

6. “Stuck” Live Migrations

• Live migrations can get stuck
• Instances left in a paused state on both ends
• Monitor socket is unpresponsive, Libvirt is helpless
• Generally a result of an overly aggressive “max donwtime” and rapidly changing memory state (e.g., JVM)
• Can be a result of a QEMU issue/bug
• manageSave (suspend) will generally be prone as well

7. Live Migration Flow

http://bodenr.blogspot.com/2014/03/openstack-nova-vm-migration-live-and.html

3. Reboot

• Soft reboot
• It relies completely on the guest OS and ACPI passed through QEMU
• Hard reboot
• Just make it work. 
• It resolves most issue
• It is at the hypervisor and Nova level 
• It makes zero assumptions about the state of the hypervisor
• Notable effort has been placed to make internal operations idempotent, and call them here.
• Steps
• Destroy the domain
• Equivalent of `virsh destroy`
• Does not destroy data, only the QEMU process
• Effectively a `kill -9` of the QEMU process
• Re-establish any and all volume connections.
• Regenerate the Libvirt XML
• Check for and re-download any missing backing files (instance_dir/_base)
• Plug VIFs (re-create bridges, VLAN interfaces, etc.)
• Regenerate and apply iptables rules