Under Development

There is an open bug in SmartOS that needs to be fixed for this all to work.
These are development notes until this header is removed.

Motivation

I (still) don't run VMware but I do have a SmartOS machine (it's a little nicer than the one from a decade ago).
I now work on Triton for my day job and I want to run CoaL for some testing.

Networking

The first trick is going to be to get some appropriate network tags set up and configured in the way that the CoaL image expects. I'm going to set up both an admin network and an "external" network. The latter will perform the same NAT that gets configured by the scripts for use with VMware.

Admin network.

This is a private network that doesn't need to reach the internet. Since I'll be confining my experiments to a single SmartOS hypervisor I'll just use an etherstub:

nictagadm add -l sdc_admin0

External network.

This one is tricker. CoaL expects this to be a network that can reach the network via NAT. We'll create another etherstub for it, then we'll create a zone to do NAT using Etherstubs:

nictagadm add -l sdc_external0

Provision a zone to be the NAT router using the following json (you can use whatever image_uuid you want, it doesn't actually matter):
coal-nat.json

{
  "alias": "coal-nat",
  "hostname": "coal-nat",
  "brand": "joyent-minimal",
  "max_physical_memory": 128,
  "image_uuid": "2f1dc911-6401-4fa4-8e9d-67ea2e39c271",
  "nics": [
    {
      "nic_tag": "external",
      "ip": "dhcp",
      "allow_ip_spoofing": "1",
      "primary": "1"
    },
    {
      "nic_tag": "sdc_external0",
      "ip": "10.88.88.2",
      "netmask": "255.255.255.0",
      "allow_ip_spoofing": "1",
      "gateway": "10.88.88.2"
    }
  ],
  "customer_metadata" : {
    "manifests" : "network/forwarding.xml\nnetwork/routing/route.xml\nnetwork/routing/ripng.xml\nnetwork/routing/legacy-routing.xml\nnetwork/ipfilter.xml\nsystem/identity.xml\n",
    "smf-import" : "mdata-get manifests | while read name; do svccfg import /lib/svc/manifest/$name; done;",
    "user-script" : "mdata-get smf-import | bash -x; echo -e 'map net0 10.88.88.0/24 -> 0/32\nrdr net0 0/0 port 22 -> 10.88.88.200 port 22 tcp' > /etc/ipf/ipnat.conf; routeadm -u -e ipv4-forwarding; svcadm enable identity:domain; svcadm enable ipfilter"
  }
}

You can also set a static IP address on the first NIC if you prefer.

Create the zone:

vmadm create -f coal-nat.json

Building the headnode VM

Normally SmartOS provides a lot of protection on the vnics. We'll be turning them all off so that the guest can do whatever it wants. This is one of the reasons I like setting up the etherstubs. Even if this VM runs amok the only other zone it can reach is that very minimal NAT zone.

We need to specify the hardcoded MAC addresses that the answers.json file is expecting to see as well:
coal-headnode.json:

{
  "alias": "coal-headnode",
  "brand": "bhyve",
  "bootrom": "uefi",
  "ram": 16384,
  "vcpus": 4,
  "autoboot": false,
  "nics": [
    {
      "mac": "00:50:56:34:60:4c",
      "nic_tag": "sdc_admin0",
      "model": "virtio",
      "ip": "dhcp",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_restricted_traffic": true,
      "allow_unfiltered_promisc": true,
      "dhcp_server": true
    },
    {
      "mac": "00:50:56:3d:a7:95",
      "nic_tag": "sdc_external0",
      "model": "virtio",
      "ip": "dhcp",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_restricted_traffic": true,
      "allow_unfiltered_promisc": true,
      "dhcp_server": true
    }
  ],
  "disks": [
    {
      "boot": true,
      "size": 8192,
      "model": "virtio"
    },
    {
      "size": 65440,
      "model": "virtio"
    }
  ]
}

Create the VM:

vmadm create -f coal-headnode.json

Copying over the CoaL USB stick image

Triton releases live at https://us-central.manta.mnx.io/Joyent_Dev/public/SmartDataCenter/triton.html
But there's a link kept up to date pointing to the latest.

curl -fLO https://us-central.manta.mnx.io/Joyent_Dev/public/SmartDataCenter/coal-latest.tgz
tar xzvf coal-latest.tgz
UUID=$(vmadm list -H -o uuid alias=coal-headnode)
qemu-img convert -f raw -O host_device coal-release-*.vmwarevm/*.img /dev/zvol/dsk/zones/${UUID?}/disk0
zfs set refreservation=0 zones/${UUID}/disk0@sdc-pristine
zfs snapshot zones/${UUID?}/disk0@sdc-pristine

Pre-configuring CoaL

We need to obtain the CoaL answers.json file and reconfigure Loader so that it will behave correctly in the VM.

lofiadm -l -a /dev/zvol/dsk/zones/${UUID?}/disk0
mount -F pcfs /devices/pseudo/lofi@2:c /mnt
curl -kL https://raw.githubusercontent.com/tritondatacenter/sdc-headnode/master/answers.json.tmpl.external | sed 's/vga/ttya/g' > /mnt/private/answers.json
# TODO: LOADER FIXES
umount /mnt
lofiadm -d /dev/lofi/2
zfs snapshot zones/${UUID?}/disk0@configured

Optional: Get a performance boost at the cost of potential VM data corruption if the host loses power:

zfs set sync=disabled zones/${UUID?}

Now you're ready to boot your VM.

vmadm start ${UUID?} ; vmadm console ${UUID?}