Motivation

I (still) don't run VMware but I do have a SmartOS machine (it's a little nicer than the one from a decade ago).
I now work on Triton for my day job and I want to run CoaL for some testing.

Networking

The first trick is going to be to get some appropriate network tags set up and configured in the way that the CoaL image expects. I'm going to set up both an admin network and an "external" network. The latter will perform the same NAT that gets configured by the scripts for use with VMware.

Admin network.

This is a private network that doesn't need to reach the internet. Since I'll be confining my experiments to a single SmartOS hypervisor I'll just use an etherstub:

nictagadm add -l sdc_admin0

External network.

This one is tricker. CoaL expects this to be a network that can reach the network via NAT. We'll create another etherstub for it, then we'll create a zone to do NAT using Etherstubs:

nictagadm add -l sdc_external0

Provision a zone to be the NAT router using the following json (you can use whatever image_uuid you want, it doesn't actually matter):
coal-nat.json

{
  "alias": "coal-nat",
  "hostname": "coal-nat",
  "brand": "joyent-minimal",
  "max_physical_memory": 128,
  "image_uuid": "2f1dc911-6401-4fa4-8e9d-67ea2e39c271",
  "nics": [
    {
      "nic_tag": "external",
      "ip": "dhcp",
      "allow_ip_spoofing": "1",
      "primary": "1"
    },
    {
      "nic_tag": "sdc_external0",
      "ip": "10.88.88.2",
      "netmask": "255.255.255.0",
      "allow_ip_spoofing": "1",
      "gateway": "10.88.88.2"
    }
  ],
  "customer_metadata" : {
    "manifests" : "network/forwarding.xml\nnetwork/routing/route.xml\nnetwork/routing/ripng.xml\nnetwork/routing/legacy-routing.xml\nnetwork/ipfilter.xml\nsystem/identity.xml\n",
    "smf-import" : "mdata-get manifests | while read name; do svccfg import /lib/svc/manifest/$name; done;",
    "user-script" : "mdata-get smf-import | bash -x; echo -e 'map net0 10.88.88.0/24 -> 0/32\nrdr net0 0/0 port 22 -> 10.88.88.200 port 22 tcp' > /etc/ipf/ipnat.conf; routeadm -u -e ipv4-forwarding; svcadm enable identity:domain; svcadm enable ipfilter"
  }
}

You can also set a static IP address on the first NIC if you prefer.

Create the zone:

vmadm create -f coal-nat.json

Building the headnode VM

Normally SmartOS provides a lot of protection on the vnics. We'll be turning them all off so that the guest can do whatever it wants. This is one of the reasons I like setting up the etherstubs. Even if this VM runs amok the only other zone it can reach is that very minimal NAT zone.

We need to specify the hardcoded MAC addresses that the answers.json file is expecting to see as well:
coal-headnode.json:

{
  "alias": "coal-headnode",
  "brand": "bhyve",
  "bootrom": "uefi",
  "ram": 16384,
  "vcpus": 4,
  "autoboot": false,
  "nics": [
    {
      "mac": "00:50:56:34:60:4c",
      "nic_tag": "sdc_admin0",
      "model": "virtio",
      "ip": "dhcp",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_restricted_traffic": true,
      "allow_unfiltered_promisc": true,
      "dhcp_server": true
    },
    {
      "mac": "00:50:56:3d:a7:95",
      "nic_tag": "sdc_external0",
      "model": "virtio",
      "ip": "dhcp",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_restricted_traffic": true,
      "allow_unfiltered_promisc": true,
      "dhcp_server": true
    }
  ],
  "disks": [
    {
      "boot": true,
      "size": 8192,
      "model": "virtio"
    },
    {
      "size": 65440,
      "model": "virtio"
    }
  ]
}

Create the VM and get the UUID:

vmadm create -f coal-headnode.json
UUID=$(vmadm list -H -o uuid alias=coal-headnode)

Copying over the CoaL USB stick image

Triton releases live at https://us-central.manta.mnx.io/Joyent_Dev/public/SmartDataCenter/triton.html

zfs set refreservation=0 zones/${UUID}/disk0

RELEASE=release-20250724-20250724T033959Z-gb8f2d08
curl -fLO https://us-central.manta.mnx.io/Joyent_Dev/public/SmartDataCenter/${RELEASE?}/headnode/usb-${RELEASE?}.tgz
tar xvf usb-${RELEASE?}.tgz usb-${RELEASE?}-8gb.img
qemu-img convert -f raw -O host_device usb-${RELEASE?}-8gb.img /dev/zvol/dsk/zones/${UUID?}/disk0

zfs snapshot zones/${UUID?}/disk0@sdc-pristine
zfs snapshot zones/${UUID?}/disk1@sdc-pristine

Pre-configuring Triton

We need to obtain the CoaL answers.json file and reconfigure Loader so that it will behave correctly in the VM.

lofiadm -l -a /dev/zvol/dsk/zones/${UUID?}/disk0
mount -F pcfs /devices/pseudo/lofi@2:c /mnt
curl -kL https://raw.githubusercontent.com/tritondatacenter/sdc-headnode/master/answers.json.tmpl.external | sed 's/vga/ttya/g' > /mnt/private/answers.json
cp /mnt/boot/loader.conf /mnt/boot/loader.conf.orig
cat /mnt/boot/loader.conf.orig | sed '/hash_name=/d;/console/s/ttyb/ttya/;/console/s/,.*text/,text/;/tty[^a]-mode/d;s/ipxe="true"/ipxe="false"/' > /mnt/boot/loader.conf
umount /mnt
lofiadm -d /dev/lofi/2
zfs snapshot zones/${UUID?}/disk0@configured

Optional: Get a performance boost at the cost of potential VM data corruption if the host loses power:

zfs set sync=disabled zones/${UUID?}

Now you're ready to boot your VM.

vmadm start ${UUID?} ; vmadm console ${UUID?}

Adding a Compute Node

For the moment I don't have a great answer to how to make the Compute Node PXE boot. This is my current workaround:

coal-computenode.json:

{
  "alias": "coal-computenode",
  "brand": "bhyve",
  "bootrom": "uefi",
  "ram": 4096,
  "vcpus": 4,
  "autoboot": false,
  "nics": [
    {
      "nic_tag": "sdc_admin0",
      "model": "virtio",
      "ip": "dhcp",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_restricted_traffic": true,
      "allow_unfiltered_promisc": true,
      "dhcp_server": true
    },
    {
      "nic_tag": "sdc_external0",
      "model": "virtio",
      "ip": "dhcp",
      "allow_dhcp_spoofing": true,
      "allow_ip_spoofing": true,
      "allow_mac_spoofing": true,
      "allow_restricted_traffic": true,
      "allow_unfiltered_promisc": true,
      "dhcp_server": true
    }
  ],
  "disks": [
    {
      "boot": true,
      "media": "cdrom",
      "path": "/ipxe/ipxe.iso",
      "model": "virtio"
    },
    {
      "size": 16000,
      "model": "virtio"
    }
  ]
}

Create the VM and get the UUID, then inject a usable ipxe ISO for netbooting:

CN_UUID=$(vmadm create -f coal-computenode.json 2>&1 |tee /dev/stderr | awk 'END{print $NF}')
mkdir -p /zones/${CN_UUID?}/root/ipxe
curl -fL -o /zones/${CN_UUID?}/root/ipxe/ipxe.iso https://raw.githubusercontent.com/tinkerbell/ipxedust/refs/heads/main/binary/ipxe.iso
vmadm start ${CN_UUID?} ; vmadm console ${CN_UUID?}

Taildrive

The Tailscale folks have shipped Taildrive (currently in alpha) and it's pretty neat. Naturally those of us using Tailscale on illumos wanted to try it out. There was nothing needed directly to get it working, but we had an indirect problem. The tailscale binary communicates with the tailscaled daemon over a unix socket, and the Tailscale folks had added some basic unix based authentication / authorization abstracted in their peercred library. That library needed support added for getpeerucred which meant I had to wire things up all the way down in x/sys/unix before then getting it into peercred. But with that work done, Taildrive now works! I tagged a release with that enabled if you're in a rush to play with it.

Using userspace-networking

Tailscale has a way to run without creating a TUN device. It means that client software on the machine can't connect directly to IPs on the Tailnet (though there is a SOCKS proxy you can use) but tailscaled can still lots of other server-y things (including Taildrive!) That's how Tailscale has been supporting AIX. Which led me to a strange realization: Tailscale had better in-tree support for AIX than it did for illumos and Solaris. No more! We are now on-par with AIX in the official tree!

What's next

I don't know if the Tailscale folks intend to ship binaries for us from their tree, but after their next release it should be possible to build illumos binaries from their tree that you could use to serve up a ZFS filesystem with Taildrive to your tailnet using the userspace-networking driver.

I will of course also rebase my TUN driver patches and tag a release as well.

Are you running Tailscale on illumos or Solaris? Let me know on Bluesky or Mastodon.

Slides:

OpenTelemetry Tracing for Dropshot

OpenTelemetry Tracing for Dropshot Nahum Shalman The QR code links to this presentation for anyone who wants to read the speaker notes or reread them later. 1 github.com/nshalman

Google Docs

Code:

[DRAFT - DO NOT MERGE] Basic OpenTelemetry integration by nshalman · Pull Request #1201 · oxidecomputer/dropshot

OpenTelemetry for DropshotThis is still very much a rough draft, but I want it to be clearly available for anyone interested.Checklist of things that are needed (note that much of it is currently…

GitHuboxidecomputer

When I read the section excerpted here, I had to immediately flip back to check the publication date. 2003. In so many ways the ongoing tragedy today is in the same place it was over 20 years ago. Rabbi Lew died in 2009. We cannot ask him what he thinks of the world today, but in many ways there is no need. Little has changed. So, to emphasize one more time, let's go back to 2003:

I think that the great philosopher George Santayana got it exactly wrong. I think it is precisely those who insist on remembering history who are doomed to repeat it. For a subject with so little substance, for something that is really little more than a set of intellectual interpretations, history can become a formidable trap— a sticky snare from which we may find it impossible to extricate ourselves. I find it impossible to read the texts of Tisha B’Av, with their great themes of exile and return, and their endless sense of longing for the land of Israel, without thinking of the current political tragedy in the Middle East. I write this at a very dark moment in the long and bleak history of that conflict. Who knows what will be happening there when you read this? But I think it’s a safe bet that whenever you do, one thing is unlikely to have changed. There will likely be a tremendous compulsion for historical vindication on both sides. Very often, I think it is precisely the impossible yearning for historical justification that makes resolution of this conflict seem so impossible. The Jews want vindication for the Holocaust, and for the two thousand years of European persecution and ostracism that preceded it; the Jews want the same Europeans who now give them moral lectures to acknowledge that this entire situation would never have come about if not for two thousand years of European bigotry, barbarism, and xenophobia. They want the world to acknowledge that Israel was attacked first, in 1948, in 1967, in 1973, and in each of the recent Intifadas. They want acknowledgment that they only took the lands from which they were attacked during these conflicts, and offered to return them on one and only one condition— the acknowledgment of their right to exist. When Anwar Sadat met that condition, the Sinai Peninsula, with its rich oil fields and burgeoning settlement towns, was returned to him. And they want acknowledgment that there are many in the Palestinian camp who truly wish to destroy them, who have used the language of peace as a ploy to buy time until they have the capacity to liquidate Israel and the Jews once and for all. They want acknowledgment that they have suffered immensely from terrorism, that a people who lost six million innocents scarcely seventy years ago should not have had to endure the murder of its innocent men, women, and children so soon again. And they want acknowledgment that in spite of all this, they stood at Camp David prepared to offer the Palestinians everything they claimed to have wanted— full statehood, a capital in East Jerusalem— and the response of the Palestinians was the second Intifada, a murderous campaign of terror and suicide bombings.

And the Palestinians? They would like the world to acknowledge that they lived in the land now called Israel for centuries, that they planted olive trees, shepherded flocks, and raised families there for hundreds of years; they would like the world to acknowledge that when they look up from their blue-roofed villages, their trees and their flowers, their fields and their flocks, they see the horrific, uninvited monolith of western culture— immense apartment complexes, shopping centers, and industrial plants on the once-bare and rocky hills where the voice of God could be heard and where Muhammad ascended to heaven. And they would like the world to acknowledge that it was essentially a European problem that was plopped into their laps at the end of the last great war, not one of their own making. They would like the world to acknowledge that there has always been a kind of arrogance attached to this problem; that it was as if the United States and England said to them, Here are the Jews, get used to them. And they would like the world to acknowledge that it is a great indignity, not to mention a significant hardship, to have been an occupied people for so long, to have had to submit to strip searches on the way to work, and intimidation on the way to the grocery store, and the constant humiliation of being subject— a humiliation rendered nearly bottomless when Israel, with the benefit of the considerable intellectual and economic resources of world Jewry, made the desert bloom, in a way they had never been able to do. And they would like the world to acknowledge that there are those in Israel who are determined never to grant them independence, who have used the language of peace as a ploy to fill the West Bank with settlement after settlement until the facts on the ground are such that an independent Palestinian state on the West Bank is an impossibility. They would like the world to acknowledge that there is no such thing as a gentle occupation— that occupation corrodes the humanity of the occupier and makes the occupied vulnerable to brutality.

And I think the need to have these things acknowledged— the need for historical affirmation— is so great on both sides that both the Israelis and the Palestinians would rather perish as peoples than give this need up. In fact, I think they both feel that they would perish as peoples precisely if they did. They would rather die than admit their own complicity in the present situation, because to make such an admission would be to acknowledge the suffering of the other and the legitimacy of the other’s complaint, and that might mean that they themselves were wrong, that they were evil, that they were bad. That might give the other an opening to annihilate or enslave them. That might make such behavior seem justifiable.

I wonder how many of us are stuck in a similar snare. I wonder how many of us are holding on very hard to some piece of personal history that is preventing us from moving on with our lives, and keeping us from those we love. I wonder how many of us cling so tenaciously to a version of a story of our lives in which we appear to be utterly blameless and innocent, that we become oblivious to the pain we have inflicted on others, no matter how unconsciously or inevitably or innocently we have have inflicted it. I wonder how many of us are terrified of acknowledging the truth of our lives because we think it will expose us. How many of us stand paralyzed between the moon and the sun; frozen — unable to act in the moment — because of our terror of the past and because of the intractability of the present circumstances that past has wrought? Forgiveness, it has been said, means giving up our hopes for a better past. This may sound like a joke, but how many of us refuse to give up our version of the past, and so find it impossible to forgive ourselves or others, impossible to act in the present?

I don't have answers. In my childhood I was promised peace in the Middle East. I am still waiting. I wish I knew what was needed to get us there.

Pirkei_Avot Chapter 5, Verse 21 implies that I am perhaps old enough to have some Wisdom, but am not yet old enough to give Counsel. The only wisdom I have obtained so far is that in most disagreements, people can disagree about the "facts", be aproaching the situation with fundamentally different values, or both. I believe that to have any meaningful discussion on a topic as fraught as this one, first common values must be established. Only then can we approach reality side-by-side, examine our beliefs, find mutually trustworthy sources of information, and find agreement about the state of reality. When values are aligned and facts are agreed upon, we might have some hope of letting go of just enough bits of history to find a path through this mess.

Decades ago I was a child promised peace. Today I have children of my own. Today on both side there are children suffering from the choices of their parents and grandparents. All the children deserve better.

There are people on both sides with genocide in their hearts. I don't yet know what to do about that, but we cannot let them win.

I wish that those who should be wise enough to provide counsel, particularly those with power, would get their acts together.

I wish for the peace and safety of all the innocents.

I wish for peace. In my lifetime. This year. Tomorrow. Or even today.

I hope that these words of Rabbi Alan Lew will reach just a few more people thanks to this post being on the internet. I hope they have touched you. Thank you for reading.

This content can also be found as part of https://github.com/tinkerbell/ipxedust/pull/88 in DifficultHardware.md.

Most modern hardware is capable of PXE booting just fine. Sometimes strange combinations of different NIC hardware / firmware connected to specific switches can misbehave.

In those situations you might want to boot into a build of iPXE but completely sidestep the PXE stack in your NIC firmware.

We already ship ipxe.iso that can be used in many situations, but most of the time that requires either an active connection from a virtual KVM client or network access from the BMC to a storage target hosting the ISO.

Some BMCs support uploading a floppy image into BMC memory and booting from that. To support that use case we have started packaging our EFI build into a bootable floppy image that can be used for this purpose.

For other projects or use cases that wish to replicate this functionality, with the appropriate versions of qemu-img, dosfstools and mtools you can build something similar yourself from upstream iPXE like so:

# create a 1440K raw disk image
qemu-img create -f raw ipxe-efi.img 1440K
# format it with an MBR and a FAT12 filesystem
mkfs.vfat --mbr=y -F 12 -n IPXE ipxe-efi.img

# Create the EFI expected directory structure
mmd -i ipxe-efi.img ::/EFI
mmd -i ipxe-efi.img ::/EFI/BOOT

# Copy ipxe.efi as the default x86_64 efi boot file
curl -LO https://boot.ipxe.org/ipxe.efi
mcopy -i ipxe-efi.img ipxe.efi ::/EFI/BOOT/BOOTX64.efi

As of writing other projects are working on automating the upload
of this floppy to a BMC.
See draft PR https://github.com/bmc-toolbox/bmclib/pull/347

code-server, Caddy, Tailscale, and Hugo = My ultimate dev environment

I think I’ve discovered my development environment equivalent to nirvana: code-server, Caddy, Tailscale, and Hugo

ChrisShort.netChris Short

Welcome back. Here's the plan:

1. Spin up an Ubuntu instance
2. Add it to my Tailnet
3. Install and run code-server
4. Install, configure, and run Caddy
5. ...
6. Profit?

When we're done we should be able to go to a browser on any machine in the Tailnet and type http://vscode (I'm assuming you've set the hostname of the new instance to "vscode") and be redirected to the full https URL which will help reassure the browser (even though HTTP over Tailscale is already secure.)

Prerequisites:

1. An existing Tailscale account and a machine on it to use as the client
2. A Tailscale Auth Key to use
3. MagicDNS needs to be enabled on your Tailnet.
4. The Tailscale HTTPS Beta feature also needs to be enabled on your Tailnet.

Here's the code:

I tested that code by using it as user-data for cloud-init, so you can go from zero to code-server over Tailscale mostly unattended.

So, spin up Ubuntu in your favorite place and either add that as the user-data or run it as root manually. When it's done you can fetch the default password with:

ssh vscode grep password: .config/code-server/config.yaml

You should be able to navigate to http://vscode (or whatever hostname you used) and get redirected to the TLS-ified URL to log in.

Check back later for more shenanigans where I'll do this inside LX branded zones on illumos!

Have you ever run ssh -D 9999 somehost?
You might find the rest of this interesting.

I'm not going to go into detail for the use cases of why you might want to use a SOCKS proxy with e.g. your web browser, but if, like me, you've ever done it, you might be interested in this method of doing something comparable with Tailscale. You'll need an existing Tailscale account and an exit node set up.

Run the daemon using userspace networking

tailscaled -tun userspace-networking -socket ~/.ssh/tailscale.socket -state ~/.ssh/tailscale.state -socks5-server localhost:9999 &

Configure it to use your exitnode and give it an identifiable hostname (you'll need to then auth the machine):

tailscale -socket ~/.ssh/tailscale.socket up --exit-node exitnode --hostname=laptop-proxy

Now you can configure your browser (I find Firefox easiest to use for this purpose) to use the SOCKS proxy just as you would have when you used SSH.

This can even be run on a machine that is already set up for Tailscale allowing use of the exit node just for this particular browser while all other network traffic from the machine behaves normally.

The tailscale.state file is sensitive; it contains the private key used for the wireguard traffic from this tailscaled process. Protect it. I put it in my .ssh directory to remind me of that.

References:

• https://tailscale.com/kb/1112/userspace-networking/
• https://tailscale.com/kb/1103/exit-nodes/

June 2022

Someone asked on Twitter and while my response was enough for them:

my ipnat.conf contains "map net0 100.64.0.0/10 -> 0/32" and "svc:/network/ipfilter:default" and "svc:/network/ipv4-forwarding:default" should be online.

... I should probably spell out the details for others.

So, notes on running an ipv4 exit node as demonstrated in a SmartOS zone:

json payload for vmadm:

{
  "alias": "exitnode-illumos",
  "hostname": "exitnode-illumos",
  "brand": "joyent",
  "max_physical_memory": 256,
  "quota": 10,
  "image_uuid": "39f99738-9a4c-11ec-8e1c-3b65b9abc26a",
  "resolvers": ["8.8.8.8","8.8.4.4"],
  "nics": [
    {
      "nic_tag": "admin",
      "ips": ["dhcp"]
    }
  ]
}

Preparatory steps to run inside the zone to setup NAT and forwarding:

echo "map net0 100.64.0.0/10 -> 0/32" > /etc/ipf/ipnat.conf
svcadm enable ipfilter
svcadm enable ipv4-forwarding

Obtain tailscale, tailscaled, and tailscale.xml and put them into place (make sure the binaries are marked executable):

cp ./tailscale{,d} /usr/local/sbin/
svccfg import tailscale.xml
mkdir -p /var/lib/tailscale
svcadm enable tailscale
tailscale up --advertise-exit-node

Authenticate the machine and mark it as allowed to be an exit node in the admin console (give it a nice ACL tag if you're feeling fancy.)

Note, the current branch has a build.sh script that you can run anywhere you have a golang toolchain (it assumes the build system is GOARCH=amd64 so if that's not the case on your build system, modify it accordingly) or as always, hit me up on IRC or DM me on Twitter and I can provide the link to my binaries.

July 2021

Work on wireguard-go has been paused. I've been working on getting Event Ports support into x/sys/unix. The code for that seems to be complete but is blocked on landing until the release of Go 1.17. See CR324630
After that lands, I can finish cleaning up my work adding Event Ports to fsnotify, though that project isn't looking very active which has me a little nervous. See fsnotify#371
Hopefully I'll find some time to pick up wireguard-go again soon, but in the meantime, last week I did a quick cleanup and rebase of my Tailscale branch. It is now based on Tailscale 1.10.1 and today I even checked in my current SMF manifests for reference: https://github.com/nshalman/tailscale/tree/illumos-1.10.1
I have been doing my builds with go 1.16.5 natively in a SmartOS zone, but because my fork of wireguard-go no longer uses cgo, it should be possible to cross compile binaries using a go 1.16.5 toolchain anywhere.
Rough instructions for a non-global zone (not fully tested; if you test or have changes let me know so that I can fix them and remove this note):

git clone https://github.com/nshalman/tailscale -b illumos-1.10.1
cd tailscale
GOOS=illumos go build ./cmd/tailscaled
GOOS=illumos go build ./cmd/tailscale
sudo cp ./tailscale{,d} /usr/local/sbin/
sudo svccfg import ./cmd/tailscaled/tailscale.xml
sudo mkdir -p /var/lib/tailscale
sudo svcadm enable tailscale
sudo tailscale up

Thanks to @richlowe for the nudge for this update. If anyone would like binaries, let me know and I'll post some and link them here.

March 2021

Alpha grade binaries of Tailscale 1.6 for illumos:
https://www.shalman.org/files/tailscale-v1.6-illumos-alpha/tailscale
https://www.shalman.org/files/tailscale-v1.6-illumos-alpha/tailscaled

There's progress in getting lots of the syscall and ioctl bits into golang's x/sys/unix and that should pave the way for a simplified patch against wireguard-go which should eventually be good enough to be upstreamed. I've rebased and rebuilt a bunch of my changes on top of Tailscale 1.6.0 and even ipv6 support seems to work. More to come.

Original Post (August 2020)

Background reading:

• Wireguard: https://www.wireguard.com/
• Tailscale: https://tailscale.com/

A while back, Josh Clulow did an initial port of wireguard-go to illumos. I was able to make some small modifications to it sufficient to get it to point the Tailscale client code at it and build tailscale binaries for illumos. Here's how to build them yourself:

You'll need git and a Go toolchain. I tested in a SmartOS zone using go114-1.14.4 and that seems to work. In the future we'll need Go 1.15.

<install git and a go toolchain>
git clone https://github.com/nshalman/wireguard-go -b tailscale-illumos wireguard-go
git clone https://github.com/nshalman/tailscale -b illumos tailscale
cd tailscale
sed -e "s|/home/admin/wireguard-go|$(cd ../wireguard-go ; pwd)|" -i go.mod
GOOS=illumos go build tailscale.com/cmd/tailscale
GOOS=illumos go build tailscale.com/cmd/tailscaled

As of the time of writing, my initial pull request to get the wireguard-go bits upstreamed is waiting on a combination of cleanups, and hopefully some additional functionality being added to x/sys/unix. Once that's done, a cleanup of my fork of tailscale to be ready for upstreaming will hopefully follow (the feature request where it's being tracked is here).

Since you shouldn't trust random people on the internet, you should probably review the code and build it yourself, however, if you know and trust me, I've uploaded binaries and my SMF manifest which you can download and experiment with (the version strings in the binaries are patched so that you know you got them from me):

• tailscaled
• tailscale
• tailscale.xml

To use them, drop the binaries into /usr/local/sbin, mkdir /etc/tailscale import them manifest (svccfg import tailscale.xml), start the daemon svcadm enable tailscale, then run tailscale up (I'm assuming you've already set up your Tailscale account, etc.)

Help is welcome and I intend to update this page as the work makes progress. Last updated August 31,2020. Feel free to troll me on Twitter if this note is still here and it's 2021. :)

On a related note, if you only need Wireguard and not Tailscale, you should also check out Mike Zeller's illumos port of boringtun which is a rust implementation. I think I managed to get it working, but stopped experimenting with it when Tailscale came out...

In mid December I received (a generous) two month's notice that I would need to find a new job. I had the luxury of being able to reach out to people I'd be interested in working with while wrapping up loose ends at work for one month, and the further luxury of being paid for another month while searching for a new job. With that much time to think about what I want to do next I came up with a framework for how to approach my search, and how to compare different opportunities should I find myself blessed with choices. What I came up with I've taken to calling a four axis model.

Important Caveats

I don't presume to be able to offer advice. I've been incredibly fortunate in so many ways in my life that I find myself standing on top of many different layers of privilege. These are things for which I can take no credit, but have made my life dramatically easier than for the average person. There's an important conversation to be had about how to make what has been possible for me possible for more people, but that is not what this post is about.

The Four Axes

These are my one-word shorthand:

1. Mission
2. Technology
3. Culture
4. Compensation

For most readers, the use of the word axis probably hints sufficiently, but in case someone else stumbles across this post, the idea here is multidimensional optimization. If all I cared about was money, I could optimize for that 4th axis by interviewing with a bunch of different companies, comparing the compensation, and simply choose the job that pays the most. Since I care about more than just money, I have to be willing to make trade-offs between these different dimensions and try to maximize the overall value that a job offer presents.

Mission

What is the company trying to accomplish in the world? What are they actually accomplishing? How does that mission connect to my values and add meaning to my day-to-day work? What story (real or imagined) will I tell myself each day about why I work where I work? Will ongoing reality re-enforce that story, or undermine it?

Technology

This is where it becomes very clear that my 4 axes are shorthand for more than 4 axes. Technology is shorthand for:

1. Do they use technologies with which I am already familiar? Will I be able to ramp up in a reasonable amount of time and be productive?
2. Will I learn and grow as a technologist?
3. Will I be able to contribute to open source projects?
4. Do they get bonus points using any of my favorite "nice to have" technologies? (For me, "nice to have" includes things like running illumos instead of Linux, using ZFS, using rust, etc.)

Culture

What are the the company's professed values? What values do their actions reveal? How do people interact within the team, across teams, across larger organizations within the company, etc. Does the company care about diversity? Is the workforce already diverse including if not particularly in upper management? Is the workplace (physical and/or digital) inclusive?

Compensation

Compensation is the value a company provides back to you in exchange for your labor. Some of it is things like an hourly rate or a base salary. Some of it is some form of equity in the company. Some of it is agreed upon explicitly at hiring time, while some of it could be a bonus based on your performance, or the company's performance, or both. Some of it could be contributing some or all of the cost of various things like insurance. Some companies offer fancy offices with catered food and snacks. Some companies will contribute to a retirement account for you. Heck, in some professions they still have pensions! Some companies will match your charitable donations. All of these things are forms of compensation, and you have to figure out how to weight each of the components of the compensation you're being offered.
I'm still relatively inexperienced with how lots of different companies try to balance their compensation packages, and I think I could have done more research during my job search. In the end, I consulted with a financial advisor to help me understand how to value base salary, public company equity, private company equity, etc. I do think it's very important to ensure that your base salary covers your critical expenses, but beyond that, I'm probably the wrong person to provide advice on compensation.

In broad strokes, however, I have started to bucket compensation into three categories:

1. Money for me for right now - base salary, insurance and other similar benefits. Additional considerations: inflation, raises, liquid equity
2. Money for me for later - Considerations: 401k matching, equity
3. Money for donating - Considerations: charitable matching, liquid equity for donating

How I use this framework

I will once again repeat the caveat that this post is not advice. I've linked some other resources at the end of this post for further reading (and if you have a favorite resource not listed, let me know about it and I'll add it!)

As a filter

Again, fully recognizing how fortunate I am that simply marking myself as "open to work" on LinkedIn provided me with a steady stream of inquiries from recruiters, this framework allowed me to, for example, politely decline some companies in industries that are just not of interest to me (even though by reputation they pay very very well.)
Similarly, I've come across companies which have pages on the their website showing the leadership team and it's ALL white men.... They might be very nice people, but chances are their workforce isn't particularly diverse.

As a tool for answering questions:

Question: "What are you looking for in your next job?"
Answer: Well, here's this framework I have in my head....

Question: "Why do you want to work at X?"
Answer: Well, here's this framework I have in my head, and here's how I think X maps onto it so far, but also, I'm still wondering about this aspect. Can you tell me more?

As a tool for asking questions:

This is where I don't know if what I'm doing is a good idea, but here it is:
In every interview where I get the chance to ask questions (most of them) I quickly run through a very brief explanation of "I'm looking to optimize across Mission, Technology, and Culture and here's what I mean by those.... Please tell me about your experience with this company along those lines."
The great thing about asking the same question of everyone is that the answers across individuals within the same company are revealing in their similarities and differences and comparing the answers across companies helps with decision making.

As a tool for making a choice:

I was fortunate to get more than one offer at roughly the same time. Having criteria for comparing offers helped me with deciding on which offer to accept.

Further Reading

• Reverse Interview
• Salary Negotiation from Patrick McKenzie
• levels.fyi has salary information about lots of companies and how they compare
• MIBEST Framework

Someone on IRC in #smartos was asking about how to turn on NAT in lx branded zones. I was pretty sure it should be possible, and found myself nerd-sniped into figuring out the exact solution.

I don't think I particularly recommend doing this, but figuring out how to do it is a good exercise in figuring out how certain things work in illumos in general, so we'll take a brief excursion into what actually happens in a joyent branded zone when you follow the steps in the wiki page I wrote forever ago. From there we'll figure out a relatively minimal set of commands to run in lx branded zones to accomplish the same thing.

The instructions list three commands to run in the firewall zone, but the third is just to verify that NAT is indeed configured, so let's look closely at the first two which do all the work:

routeadm -u -e ipv4-forwarding
svcadm enable ipfilter

The first one is turning on packet forwarding for ipv4, and the second turns on ipf filtering and NAT. But what do those commands do under the hood and can we replicate that in the lx zone?

If we look at the routeadm man page it helpfully points out that those features are also represented as SMF services which you could manage directly with svcadm and the EXAMPLES section even suggests that we could turn on ipv4 forwarding using svcadm enable ipv4-forwarding.

Once we have an SMF service, we can see how it works by looking at the service manifest. When I was poking around I did this live in a zone by running svccfg export ipv4-forwarding but for the ease of just clicking links, the source for what you'd see there is in /usr/src/cmd/cmd-inet/usr.sbin/routeadm/forwarding.xml.

Narrowing in on the start method which wants to run /lib/svc/method/svc-forwarding %m ipv4 we see that there's a helper script we need to read. Again, I looked at it at that path on my system but the source is in /usr/src/cmd/cmd-inet/usr.sbin/routeadm/svc-forwarding.
In there we find the critical line of /usr/sbin/ipadm set-prop -p forwarding=on $proto

What have we learned so far? routeadm behind the scene enables the ipv4-forwarding SMF service that in turn runs an ipadm command.

So now what about that ipfilter service?
Again, on a live system we can run svccfg export ipfilter to find the start method (source in /usr/src/cmd/ipf/svc/ipfilter.xml) and we find that it calls /lib/svc/method/ipfilter (source: /usr/src/cmd/ipf/svc/ipfilter).

If you read through this script you find some places where it calls out to ipf and ipnat whose manpages can be used to figure out what those various commands do.

Through reading the script and the man page and some experimentation I narrowed down a minimal set of steps of:

ipf -E    # enable IPF
ipnat -CF # Delete all existing rules and active mappings
ipnat -f ${ipnat_config_file:?} # load NAT rules from a file

There's one last wrinkle that tripped me up and wasted some of my time. I blithely copy-and-pasted the ipnat.conf content, ran my commands, and wondered why the packets weren't flowing. The lx brand names network interfaces with a more linux-y eth0, eth1, etc. rather than net0, net1, etc.

So, let's put this all together:

firewall-lx.json:

{
  "alias": "firewall-lx",
  "hostname": "firewall-lx",
  "brand": "lx",
  "kernel_version": "3.10.0",
  "max_physical_memory": 512,
  "image_uuid": "63d6e664-3f1f-11e8-aef6-a3120cf8dd9d",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp",
      "allow_ip_spoofing": "1"
    },
    {
      "nic_tag": "stub0",
      "ip": "10.0.0.1",
      "netmask": "255.255.255.0",
      "allow_ip_spoofing": "1",
      "primary": "1"
    }
  ]
}

client.json:

{
  "alias": "client",
  "hostname": "client",
  "brand": "joyent-minimal",
  "max_physical_memory": 256,
  "image_uuid": "cfa9c88e-03f8-11eb-9980-879ff7980a9f",
  "nics": [
    {
      "nic_tag": "stub0",
      "ip": "10.0.0.2",
      "netmask": "255.255.255.0",
      "gateway": "10.0.0.1",
      "primary": "1"
    }
  ]
}

On the host:

nictagadm add -l stub0
vmadm create -f client.json

In a separate window, zlogin into the client zone and start pinging an external ip with ping -ns e.g. ping -ns 8.8.8.8
So far no packets should be flowing, let's fix that.

Back out on the host

vmadm create -f firewall-lx.json

Now zlogin into your lx branded "firewall" zone and let's get packets flowing (note how we invoke the native illumos tools from under /native):

echo "map eth0 10.0.0.2/32 -> 0/32" > /etc/ipnat.conf
/native/usr/sbin/ipadm set-prop -p forwarding=on ipv4
/native/usr/sbin/ipf -E
/native/usr/sbin/ipnat -CF
/native/usr/sbin/ipnat -f /etc/ipnat.conf

At this point, your client zone should start seeing ping replies.
Whether to use this in production and how to get it to survive reboots of the "firewall" zone is left as an exercise for the reader.

First, prepare the receiving end by creating the pool. If this is not a bootable pool, just create it the simple way:

poolname=rpool
disk=/dev/sda
zpool create ${poolname:?} ${disk:?}

Now start up mbuffer to receive the initial send stream ("-s" is for resumable receive in case the receive gets interrupted part way through)
I'm using a block size of 128k, a buffer size of 1G and listening on port 9090:

mbuffer -s 128k -m 1G -I 9090 | zfs receive -Fs ${poolname:?}

Now switch over to the sending machine. Here we will take a snapshot, and then send it over mbuffer to the receving machine. We'll do a raw and resumable send.

poolname=rpool
snapname=migration-snap1
receiver=receiver.example.com
zfs snapshot -r ${poolname:?}@${snapname:?}
zfs send -Rvw ${poolname:?}@${snapname:?} | mbuffer -s 128k -m 1G -O ${receiver:?}:9090

If the sending side has changed since the first snapshot completed sending, do an incremental send/receive to sync up the last changes (repeat as needed)

On receving machine, rerun the receiving command:

mbuffer -s 128k -m 1G -I 9090 | zfs receive -Fs ${poolname:?}

On the sending machine, take another snapshot and send over the incremental stream:

snap2=migration-snap2
zfs snapshot -r ${poolname:?}@${snap2:?}
zfs send -Rvw ${poolname:?}@${snapname:?} ${poolname:?}@${snap2:?} | mbuffer -s 128k -m 1G -O ${receiver:?}:9090

References:

• https://web.archive.org/web/20130922023108/https://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/
• https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html
• https://openzfs.github.io/openzfs-docs/man/8/zfs-receive.8.html

Some notes I took down as I stood up my new zone hosted by https://mnx.io.

Pkgsrc branch switch

pkgin up
pkgin fug
curl -LO https://pkgsrc.joyent.com/packages/SmartOS/bootstrap-upgrade/bootstrap-2019Q2-x86_64-upgrade.tar.gz
curl -LO https://pkgsrc.joyent.com/packages/SmartOS/bootstrap-upgrade/bootstrap-2019Q2-x86_64-upgrade.tar.gz.asc
curl -sS https://pkgsrc.joyent.com/pgp/DE817B8E.asc | gpg --import
gpg --verify bootstrap-2019Q2-x86_64-upgrade.tar.gz.asc
PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2019Q2/x86_64/All pkg_add -U pkg_install pkgin
gtar -xzvf bootstrap-2019Q2-x86_64-upgrade.tar.gz -C /
pkgin full-upgrade
rm bootstrap-2019Q2-x86_64-upgrade.tar.gz*
sm-set-hostname shalman-20191007

Reference: https://pkgsrc.joyent.com/install-on-illumos/#64bit-upgrade

Salt

pkgin in salt
mkdir -p /srv/spm/salt
# put stuff into place under /srv/spm/salt
salt-call --local state.apply

Hurricane Electric Tunnel

dladm create-iptun -T ipv4 -a \
  local=67.158.54.193,remote=184.105.253.14 v4_he0
ipadm create-addr -T static -a \
  local=2001:470:1f10:251::2,remote=2001:470:1f10:251::1 v4_he0/v6
route -p add -inet6 default 2001:470:1f10:251::1

Reference: https://blog.brianewell.com/hurricane-electric-on-smartos/

SSL related updates

• https://ssl-config.mozilla.org/
• https://letsencrypt.org/docs/caa/
• https://www.ssllabs.com/ssltest/index.html

Just a heads-up that I have to migrate my domain off of the JPC and there will probably be a downtime of my blog and other services for an unknown duration (though I'll do my best to keep it as short as possible) to complete the migration.

It might also take a while for me to restore IPv6 connectivity.

If you come across any broken links after I complete the migration, let me know.

Update: If you can see this message, clearly migration is nearly complete. This blog has been upgraded to the latest version of Ghost, and I'm redirecting all traffic to HTTPs.

Motivation

There are a lot of blog posts and wiki pages about how to set up Wireguard, but I still had to do a bunch of trial and error to come up with a configuration that worked for me. I have two goals:

1. Secure all traffic from my Android phone for privacy / when on unsecured WiFi.
2. Full access to my home network resources when away from home.

Caveats:

• This does client key generation on the server, which is fine for this simple use case, but not a cryptographic best practice.
• This sets up only a single client. If you need multiple clients, you'll have to get fancier.
• As part of meeting my goals, ALL phone traffic goes through the VPN. That may not be your desired configuration.

Preparation

You will need:

1. The public IP address of your router (or a DNS record that points to it)
2. An open port on your router forwarded to wherever you run Wireguard
3. The IP address of the DNS server your phone should use when connected to your home network (maybe your router, maybe not)
4. Two addresses in a private subnet not used elsewhere on your home network

Implementation

I hosted all of this in a VM running Ubuntu 18.04.1.
At the end of this process, a QR Code is displayed to be scanned by the Wireguard Android app.
This script is run as root sudo bash -x simple-wireguard.sh:

#!/bin/bash -x

################ CHANGE THESE ##################
# Your Home IP or DNS record
WG_ADDRESS=wireguard.your.domain
# UDP port forwarded to this machine
WG_PORT=12345
# DNS entries for the client to use when on VPN
WG_DNS=192.168.1.1,8.8.8.8
# Unused private IPs for this connection
WG_SERVER_INT=172.16.17.1
WG_CLIENT_INT=172.16.17.2
################################################

add-apt-repository -y ppa:wireguard/wireguard
apt-get -y update
apt-get -y install wireguard qrencode

# Enable Packet Forwarding
echo net.ipv4.ip_forward=1 > /etc/sysctl.d/wireguard.conf
sysctl -p /etc/sysctl.d/wireguard.conf

# Generate Keys and Configurations
cd /etc/wireguard
wg genkey | tee server.private | wg pubkey > server.public
wg genkey | tee client.private | wg pubkey > client.public

# Server Configuration sets up NAT
cat >wg0.conf <<EOF
[Interface]
Address = ${WG_SERVER_INT}
SaveConfig = false
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o net0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o net0 -j MASQUERADE
ListenPort = ${WG_PORT}
PrivateKey = $(cat server.private)

[Peer]
PublicKey = $(cat client.public)
AllowedIPs = ${WG_CLIENT_INT}
EOF

cat >client.conf <<EOF
[Interface]
Address = ${WG_CLIENT_INT}
PrivateKey = $(cat client.private)
DNS = ${WG_DNS}

[Peer]
PublicKey = $(cat server.public)
Endpoint = ${WG_ADDRESS}:${WG_PORT}
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 10
EOF

# Turn on and enable for boot
systemctl start wg-quick@wg0
systemctl enable wg-quick@wg0

# Show the QR Code
qrencode -t ansiutf8 < client.conf

Better Tools

I wanted something simple and comprehensible that I built myself, but there are tools out there that build fancier configurations that are recommended by smart people in this space:

• https://github.com/trailofbits/algo
• https://github.com/StreisandEffect/streisand

Another option that is built atop wireguard-go is Tailscale. Their service is proprietary, though many of their client implementations are open-source.

References

• https://www.wireguard.com/quickstart/
• https://www.stavros.io/posts/how-to-configure-wireguard/
• https://wiki.debian.org/Wireguard
• https://www.ckn.io/blog/2017/11/14/wireguard-vpn-typical-setup/
• https://emanuelduss.ch/2018/09/wireguard-vpn-road-warrior-setup/
• https://github.com/michaeljs1990/docs/blob/master/software/wireguard.md

Below are a couple of gists I worked on when helping someone else trying to use the code above:

Message from 2018: I was going through my blog post drafts and found this post.
I've made few small tweaks to it that seem to be what I was hoping to add before publishing.
If you've ever wanted to replace your SSH access to a native branded zone with docker exec access, this is the blog post for you. We now return you to the bleeding edge of at least 1 or 2 years ago:

As a quick pre-emptive caveat, this post describes using the Docker CLI tool basically to managed what Joyent are calling "infrastructure containers". Classic illumos zones with many processes in them, not the kind of containers that you typically create and run with Docker. It's a how-to on using the Docker CLI instead of the CloudAPI tools you might usually use in the Joyent Public Cloud (JPC).

I stumbled across this gem in the Joyent docs the other day. It turns out that you can use the Docker CLI tool to create and manage "joyent-minimal" branded zones. Getting Docker set up with your JPC account is beyond the scope of this blog post but it's covered here.

As that first link describes, if instead of a regular image name you provide the uuid of a "smartos" type image it will be used to provision a joyent-minimal branded zone.

Let's fire up a recent image with 128 MB of RAM and a public IP:

docker run -P -d -m 128 --name=tiny 390639d4-f146-11e7-9280-37ae5c6d53d4 /sbin/init

It will sidestep all the normal zone setup, so we have to manually set the default route:

docker exec -it tiny /bin/bash -c 'route -p add default $(mdata-get sdc:nics | json 1.gateway)'

Thanks to changes I worked on a while back you can import pretty much any service manifest you want quite easily to bring various services online.

Instead of SSH access, use the docker cli to log in to your container:

docker exec -it tiny /bin/bash

Experiment with installing packages and enabling any necessary SMF services; have fun!

And of course, don't forget to delete this zone when you're done with it:

docker rm -f tiny