Silly SmartOS hack: ipf/ipnat in lx zones

Someone on IRC in #smartos was asking about how to turn on NAT in lx branded zones. I was pretty sure it should be possible, and found myself nerd-sniped into figuring out the exact solution.

I don't think I particularly recommend doing this, but figuring out how to do it is a good exercise in figuring out how certain things work in illumos in general, so we'll take a brief excursion into what actually happens in a joyent branded zone when you follow the steps in the wiki page I wrote forever ago. From there we'll figure out a relatively minimal set of commands to run in lx branded zones to accomplish the same thing.

The instructions list three commands to run in the firewall zone, but the third is just to verify that NAT is indeed configured, so let's look closely at the first two which do all the work:

routeadm -u -e ipv4-forwarding
svcadm enable ipfilter

The first one is turning on packet forwarding for ipv4, and the second turns on ipf filtering and NAT. But what do those commands do under the hood and can we replicate that in the lx zone?

If we look at the routeadm man page it helpfully points out that those features are also represented as SMF services which you could manage directly with svcadm and the EXAMPLES section even suggests that we could turn on ipv4 forwarding using svcadm enable ipv4-forwarding.

Once we have an SMF service, we can see how it works by looking at the service manifest. When I was poking around I did this live in a zone by running svccfg export ipv4-forwarding but for the ease of just clicking links, the source for what you'd see there is in /usr/src/cmd/cmd-inet/usr.sbin/routeadm/forwarding.xml.

Narrowing in on the start method which wants to run /lib/svc/method/svc-forwarding %m ipv4 we see that there's a helper script we need to read. Again, I looked at it at that path on my system but the source is in /usr/src/cmd/cmd-inet/usr.sbin/routeadm/svc-forwarding.
In there we find the critical line of /usr/sbin/ipadm set-prop -p forwarding=on $proto

What have we learned so far? routeadm behind the scene enables the ipv4-forwarding SMF service that in turn runs an ipadm command.

So now what about that ipfilter service?
Again, on a live system we can run svccfg export ipfilter to find the start method (source in /usr/src/cmd/ipf/svc/ipfilter.xml) and we find that it calls /lib/svc/method/ipfilter (source: /usr/src/cmd/ipf/svc/ipfilter).

If you read through this script you find some places where it calls out to ipf and ipnat whose manpages can be used to figure out what those various commands do.

Through reading the script and the man page and some experimentation I narrowed down a minimal set of steps of:

ipf -E    # enable IPF
ipnat -CF # Delete all existing rules and active mappings
ipnat -f ${ipnat_config_file:?} # load NAT rules from a file

There's one last wrinkle that tripped me up and wasted some of my time. I blithely copy-and-pasted the ipnat.conf content, ran my commands, and wondered why the packets weren't flowing. The lx brand names network interfaces with a more linux-y eth0, eth1, etc. rather than net0, net1, etc.

So, let's put this all together:

firewall-lx.json:

{
  "alias": "firewall-lx",
  "hostname": "firewall-lx",
  "brand": "lx",
  "kernel_version": "3.10.0",
  "max_physical_memory": 512,
  "image_uuid": "63d6e664-3f1f-11e8-aef6-a3120cf8dd9d",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "dhcp",
      "allow_ip_spoofing": "1"
    },
    {
      "nic_tag": "stub0",
      "ip": "10.0.0.1",
      "netmask": "255.255.255.0",
      "allow_ip_spoofing": "1",
      "primary": "1"
    }
  ]
}

client.json:

{
  "alias": "client",
  "hostname": "client",
  "brand": "joyent-minimal",
  "max_physical_memory": 256,
  "image_uuid": "cfa9c88e-03f8-11eb-9980-879ff7980a9f",
  "nics": [
    {
      "nic_tag": "stub0",
      "ip": "10.0.0.2",
      "netmask": "255.255.255.0",
      "gateway": "10.0.0.1",
      "primary": "1"
    }
  ]
}

On the host:

nictagadm add -l stub0
vmadm create -f client.json

In a separate window, zlogin into the client zone and start pinging an external ip with ping -ns e.g. ping -ns 8.8.8.8
So far no packets should be flowing, let's fix that.

Back out on the host

vmadm create -f firewall-lx.json

Now zlogin into your lx branded "firewall" zone and let's get packets flowing (note how we invoke the native illumos tools from under /native):

echo "map eth0 10.0.0.2/32 -> 0/32" > /etc/ipnat.conf
/native/usr/sbin/ipadm set-prop -p forwarding=on ipv4
/native/usr/sbin/ipf -E
/native/usr/sbin/ipnat -CF
/native/usr/sbin/ipnat -f /etc/ipnat.conf

At this point, your client zone should start seeing ping replies.
Whether to use this in production and how to get it to survive reboots of the "firewall" zone is left as an exercise for the reader.