Deploy vCenter Server Appliance with Terraform

I’m a big fan of William Lam’s scripts to deploy nested vSphere lab environments. A few months ago, he published a script for vSphere 7.0 that deploys the usual suspects. It also deploys vSphere with Kubernetes. I’ve always wanted to see if I could build something similar using Terraform. I think it would be awesome to run a terraform apply and go from an OVA file to a fully deployed vCenter.

I’ve messed around with this in the past. I even fixed a bug that made the VCSA as well as William’s Nested ESXi Appliances undeployable. Without any hacks you could deploy a VCSA and get it online enough for network connectivity. Due to limitations, you would have to manually configure the VCSA from the VAMI interface. As another option, you could manually edit the VCSA OVF file and change all of the properties marked userConfigurable=false to userConfigurable=true. The last snag was that the Terraform provider didn’t support direct deployment from an OVA. You had to manually import the image and mark it a template.

William blogged somewhat recently that OVA/OVF support was quite a bit more robust. With this in mind, I thought I would take a look at this again. I played around a bit and just submitted a PR to implement some functionality similar to VMware’s ovftool that allows fields with userConfigurable=false to be set anyways. If my PR makes it upstream, you can easily deploy a vCenter Server Appliance from an OVA file. It will be fully up and running using only Terraform! It took around 22 minutes in my lab environment for this to work. Specifically, it is running on the NAS I built recently.

Stay tuned, I hope to have a full lab environment working in the near future.

Terraform code to deploy a vCenter Server Appliance

resource "vsphere_virtual_machine" "vcsa" {
    name = "vcsa"

    num_cpus = 2
    memory   = 12288
    
    datacenter_id    = data.vsphere_datacenter.dc.id
    folder           = var.folder_name
    resource_pool_id = vsphere_resource_pool.resource_pool.id
    host_system_id   = data.vsphere_host.host.id
    datastore_id     = data.vsphere_datastore.datastore.id

    cpu_hot_add_enabled    = true
    cpu_hot_remove_enabled = true
    memory_hot_add_enabled = true

    ovf_deploy {
        local_ovf_path    = "./ova/VMware-vCenter-Server-Appliance-7.0.0.10300-16189094_OVF10.ova"
        disk_provisioning = "thin"
        ip_protocol       = "IPv4"
        ovf_network_map   = {
            "Network 1" = data.vsphere_network.network.id
        }

        // this isn't upstream yet
        enable_hidden_properties = true
    }

    vapp {
        properties = {
            "guestinfo.cis.appliance.net.addr.family" = "ipv4"
            "guestinfo.cis.appliance.net.mode"        = "static"
            "guestinfo.cis.appliance.net.addr"        = "192.168.1.100"
            "guestinfo.cis.appliance.net.dns.servers" = "192.168.1.1"
            "guestinfo.cis.appliance.net.prefix"      = 24
            "guestinfo.cis.appliance.net.gateway"     = "192.168.1.1"
            "guestinfo.cis.appliance.net.pnid"        = "vcsa.example.com"
            "guestinfo.cis.appliance.root.passwd"     = "hunter2"
            "guestinfo.cis.vmdir.password"            = "hunter2"
            "guestinfo.cis.ceip_enabled"              = "False"

            // these can't be set with upstream (yet)
            "guestinfo.cis.appliance.ssh.enabled" = "True"
            "guestinfo.cis.deployment.autoconfig" = "True"
            "guestinfo.cis.appliance.ntp.servers" = "pool.ntp.org"
            "guestinfo.cis.vmdir.domain-name"     = "vsphere.local"
            "guestinfo.cis.vmdir.username"        = "[email protected]"
        }
    }

    provisioner "local-exec" {
        // without a provisioner, the deployment would return once VMware tools sees an IP address
        // this waits for the appliance system health api to return green
        command = "${path.module}/vcsa-provisioner.sh"
        environment = {
            VCENTER_HOSTNAME = "vcsa.example.com"
            VCENTER_USERNAME = "[email protected]"
            VCENTER_PASSWORD = "hunter2"
            TIMEOUT          = 60
        }
    }

    lifecycle {
        ignore_changes = [
            // it looks like some of the properties get deleted from the VM after it is deployed
            // just ignore them after the initial deployment
            vapp,
        ]
    }
}

The shell script in the provisioner is fairly simple, just try to log into the appliance health REST API every minute and check the status:

#!/bin/bash

i=0
while true; do
    if [ $i -le $TIMEOUT ]; then
        echo "Attempting to get vCenter Auth Token"
        if TOKEN=`curl -s -k -u ${VCENTER_USERNAME}:${VCENTER_PASSWORD} -X POST https://${VCENTER_HOSTNAME}/rest/com/vmware/cis/session | jq -r .value`; then
            echo "Attempting to get appliance system health status"
            
            if HEALTH=`curl -s -k -H "vmware-api-session-id: ${TOKEN}" -X GET https://${VCENTER_HOSTNAME}/rest/appliance/health/system | jq -r .value`; then
                if [ "${HEALTH}" = "green" ]; then break; fi
            fi
        fi
        echo "Unable to verify vCenter status, sleeping 1 minute"
        sleep 1m
        ((i++))
    else
        echo "Provisioner took over ${TIMEOUT} minutes.  Exiting with error"
        exit 1
    fi
done
    
exit 0