Motivation

I don't have any machines that run VMware but I do have a nice SmartOS machine.
I have some SDC features I want to test. I want to test my own changes to SDC to make sure that they are safe, and I want to play with the latest sdc-docker features in a sandbox.

Networking

The first trick is going to be to get some appropriate network tags set up and configured in the way that the CoaL image expects. I'm going to set up both an admin network and an "external" network. The latter will perform the same NAT that gets configured for VMware by the scripts provided by Joyent.

Admin network.

This is a private network that doesn't need to reach the internet. Since I'll be confining my experiments to a single SmartOS hypervisor I'll just use an etherstub:

nictagadm add -l sdc_admin0

External network.

This one is tricker. CoaL expects this to be a network that can reach the network via NAT. We'll create another etherstub for it, then we'll create a zone to do NAT using Etherstubs:

nictagadm add -l sdc_external0

Provision a zone to be the NAT router using the following json (you can use whatever image_uuid you want, it doesn't actually matter):

{
  "alias": "coal-nat",
  "hostname": "coal-nat",
  "brand": "joyent",
  "max_physical_memory": 128,
  "image_uuid": "62f148f8-6e84-11e4-82c5-efca60348b9f",
  "nics": [
    {
      "nic_tag": "external",
      "ip": "dhcp",
      "allow_ip_spoofing": "1",
      "primary": "1"
    },
    {
      "nic_tag": "sdc_external0",
      "ip": "10.88.88.2",
      "netmask": "255.255.255.0",
      "allow_ip_spoofing": "1",
      "gateway": "10.88.88.2"
    }
  ],
  "customer_metadata" : {
    "user-script" : "echo 'map net0 10.88.88.0/24 -> 0/32' > /etc/ipf/ipnat.conf; routeadm -u -e ipv4-forwarding; svcadm enable ipfilter; svcadm disable ssh"
  }
}

You can also set a static IP address on the first NIC if you prefer.

Building the headnode VM

Normally SmartOS provides a lot of protection on the vnics. We'll be turning them all off so that the guest can do whatever it wants. This is one of the reasons I like setting up the etherstubs. Even if this VM runs amok the only other zone it can reach is that very minimal NAT zone.

We need to specify the hardcoded MAC addresses that the answers.json file is expecting to see as well:

{
  "alias": "coal-headnode",
  "brand": "kvm",
  "ram": 8192,
  "vcpus": 3,
  "autoboot": false,
  "nics": [
    {
      "mac": "00:50:56:34:60:4c",
      "nic_tag": "sdc_admin0",
      "model": "virtio",
      "ip": "dhcp",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_restricted_traffic": true,
      "allow_unfiltered_promisc": true,
      "dhcp_server": true
    },
    {
      "mac": "00:50:56:3d:a7:95",
      "nic_tag": "sdc_external0",
      "model": "virtio",
      "ip": "dhcp",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_restricted_traffic": true,
      "allow_unfiltered_promisc": true,
      "dhcp_server": true
    }
  ],
  "disks": [
    {
      "size": 4096,
      "model": "virtio"
    },
    {
      "size": 61440,
      "model": "virtio"
    }
  ]
}

Create the VM:

vmadm create -f coal-headnode.json

Copying over the CoaL USB stick image

wget http://us-east.manta.joyent.com/Joyent_Dev/public/SmartDataCenter/coal-latest.tgz
tar xzvf coal-latest.tgz
UUID=$(vmadm list -H -o uuid alias=coal-headnode)
qemu-img convert -f raw -O host_device coal-release-*-4gb.vmwarevm/4gb.img /dev/zvol/dsk/zones/${UUID}-disk0
zfs snapshot zones/${UUID}-disk0@sdc-pristine

Pre-configuring CoaL

We need to obtain the CoaL answers.json file and reconfigure GRUB so that it will behave correctly in the VM.

mount -F pcfs /dev/zvol/dsk/zones/${UUID}-disk0:c /mnt
curl -kL https://raw.githubusercontent.com/joyent/sdc-headnode/master/answers.json.tmpl.external | sed 's/vga/ttya/g' > /mnt/private/answers.json
sed -e 's/ttyb/ttya/;s/1,0,2,3/0,1,2,3/;s/default 0/default 1/' -i.smartosbak /mnt/boot/grub/menu.lst*
umount /mnt
zfs snapshot zones/${UUID}-disk0@sdc-fixed

Optional: Get a performance boost at the cost of potential VM data corruption if the host loses power:
zfs set sync=disabled zones/${UUID}-disk1

Now you're ready to boot your VM.

vmadm start ${UUID} ; vmadm console ${UUID}

EDIT April 10,2017:

This blog post used to specify ide for the disk model and e1000 for the NIC model. I have since switched it to use virtio for both for performance reasons.
I also added the note about making ZFS process the ZVOL faster when I made those changes.