Overriding Driver Config Files on SmartOS

First, if you haven't already read it, you should at least skim Keith Wesolowski's post on anonymous tracing which discusses the underlying feature in SmartOS that enables this.

(Aside: If my GRUB configs look different than yours, it might be because you're using the real USB stick while I'm using a ZFS formatted one that I made using the GRUB configs from the ISO... If there's demand I can do a writeup of how I build and maintain ZFS formatted SmartOS boot media...)

Okay, so you have your USB stick or PXE server serving up SmartOS to some machine that has a driver that needs to be tweaked.

Maybe you bought some questionable SSDs that lie about their block size and you want to modify sd.conf to fix that.

Maybe like me you have a server with too many ethernet cards in it (long story, don't ask, we should probably all just be buying Pluribus gear, but since Pluribus doesn't list prices I'm pretty sure I can't afford it....)

Okay, so maybe you're modifying kernel/drv/sd.conf. I happen to be modifying kernel/drv/ixgbe.conf.

I figured out which details I wanted to tweak in that file and I put it in the root of my boot media next to the platform directory in a file. As Keith explained, the file name of this file isn't important, it will get dropped into the file path/name you specify in GRUB, but I kept it the same to prevent confusion.

Here's what the relevant menu.lst section looked like before I made any changes:

title Live 64-bit (text)
   kernel /platform/i86pc/kernel/amd64/unix -B console=text,smartos=true
   module /platform/i86pc/amd64/boot_archive

Now here's my changed version (note the addition of type=rootfs name=ramdisk to the line with the boot_archive. That part is really important.):

title Live 64-bit (text)
   kernel /platform/i86pc/kernel/amd64/unix -B console=text,smartos=true
   module /platform/i86pc/amd64/boot_archive type=rootfs name=ramdisk
   module /ixgbe.conf type=file name=kernel/drv/ixgbe.conf

When I boot my system with this modified GRUB configuration I can see the content of the ixgbe.conf file from my boot media in /system/boot/<the name specified in menu.lst> in my case /system/boot/kernel/drv/ixgbe.conf but even more importantly, when the kernel loaded the ixgbe driver, it read that file and loaded in those changes.

Now I can boot my goofy server without building my own customized boot archive with my change in it. This is particularly great because now I can upgrade that server to the latest version of the platform at any time simply by downloading the latest version from Joyent.