1. Libvirt Python API

  • Compute manager calls the driver
    • `self.driven.pause(instance)`
      • It will suspend the instance
  • Nova Libvirt Driver

2. Libvirt Domain 

  • Libvirt domains are defined via XML
  • Domains defined by Nova are persistent
  • XML is re-generated on Hard Reboot
    • Manual costomerizations will be overriden
  • Logging occurs per-domain
    • /var/log/libvirt/qemu/instance_name.log

3. Virsh 

  • virsh is the command line tool for Libvirt
    • Consumes the same API referenced earlier via C
  • virsh list
    • List all domains by name
  • virsh domname uuid
    • Returns the instance_name
  • virsh dumpxml uuid
    • Configuration for an individual domain

4.  Libvirt Domain XML

  • Translated by Libvirt to ultimately call QEMU with the right arguments
  • The XML generated by Nova ends up being modified, fairly heavility, by Libvirt upon definition
  • Key difference between “active” and “inactive” XML
    • Numerous values are derived at time of instance start
    • Output from an active “dumpxml” will likely fail to define
      • Use `virsh dumpxml –inactive uuid `

    5. Libvirtd Configuration

    • Critically important information tuning for Libvirt 1.1.x and newer, example
      • max_clients = 50

    Live Migration

    1. Life Migration Workflow

    • Verify the storage backend is appropriate for the migration type
      • Perform a shared storage check for normal migrations
      • Do the inverse for block migrations
      • Checks are run on both the source and destination, orchestrated via RPC calls from the scheduler
    • On the destination
      • Create the necessary volume connections
      • If block migration, create the instance directory, populate missing backing files from Glance and create empty instance disks
    • On the source
      • Initiate the actual live migration
    • Upon complete
      • Generate the Libvirt XML and define it on the destination

      2. Migrations

      • Why migration
        • Operations
          • Key to performing non-distruptive work
          • Re-balancing workloads and resources
        • Expectations versus reality
          • Special snowflakes
          • Ephemeral instance and the “cloud” way
      • Type of migration
        • Migrate
          • Completely “cold”, libvirt does almost nothing
          • Share code path with “resize”
          • Extremely brittle (users SSh and copies files around)
        • Live migration
          • Orchestrated almost entirely by Libvirt (via DomainMigrateToURI)
        • Block migration
          • Similar code path as live migraiton
          • More risky and brittle (disks are moving along with state)

      3. Live Migrations

      • Nova offloads capabilities comparisons to Libvirt
        • The API equivalent of virsh capabilities is run by the scheduler on the source and destination; 
      • Nova live migraiton
        • Important config options
          • Live_migration_flat =+ VIR_MIGRATE_LIVE
          • block_migration_flag=+ VIR_MIGRATE_LIVE
        • Standardized virtual CPU flags
          • libvirt_cpu_model = custome
          • libvirt_cpu_model = cpu64-rhel6
        • “Max Downtime” (not currently tunable)
          • Look for upstream patches soon
          • Qemu will keep doing when the cut can be done in “30” millseconds

      4. Brittle Operations

      • Any long running, synchronous tasks
        • All migrations (memory sync, disk sync, etc)
      • No graceful way to stop services
      • Most prone to failure
        • Migrate and resize
        • Live migraiton (block or otherwise)
        • Instance snapshot

      5. Recovering from failures

      • Always investigate before forcing actions
        • Look at the log for excpetions
        • Check whether an instance is running on multiple hypervisors
        • Nova reset-state –active and `nova reboot –hard can go a long way
      • Sometime, brute force is going to be required
        • Kill -9 qumu or kvm processes
        • After the database records, commonly `host`

      6. “Stuck” Live Migrations

      • Live migrations can get stuck
      • Instances left in a paused state on both ends
        • Monitor socket is unpresponsive, Libvirt is helpless
      • Generally a result of an overly aggressive “max donwtime” and rapidly changing memory state (e.g., JVM)
      • Can be a result of a QEMU issue/bug
        • manageSave (suspend) will generally be prone as well

      OpenStack Overview

      1. Architecture

      2. Commands

      • In compute node, run `virsh capabilities`, you can see the capability of that node.
      • `virsh dumpxml instance-id`
        • Describe the vm

      3. Reboot

      • Soft reboot
        • It relies completely on the guest OS and ACPI passed through QEMU
      • Hard reboot
        • Just make it work. 
        • It resolves most issue
        • It is at the hypervisor and Nova level 
        • It makes zero assumptions about the state of the hypervisor
          • Notable effort has been placed to make internal operations idempotent, and call them here.
        • Steps
          • Destroy the domain
            • Equivalent of `virsh destroy`
            • Does not destroy data, only the QEMU process
            • Effectively a `kill -9` of the QEMU process
          • Re-establish any and all volume connections.
          • Regenerate the Libvirt XML
          • Check for and re-download any missing backing files (instance_dir/_base)
          • Plug VIFs (re-create bridges, VLAN interfaces, etc.)
          • Regenerate and apply iptables rules