Exploring epoll on SmartOS
Introduction
Back in April, Joyent added an implementation of the epoll API to their downstream tree of illumos. This is interesting because it means that many otherwise portable applications that depend on this (formerly) Linux-specific API can now be built and run on SmartOS. Rather than patch every piece of software that depends on epoll to support event ports (the native API in illumos that provides a similar feature) we can simply use the unmodified software.
How well does it work? Let's try it out!
Sample Application: ngircd
A couple years ago I decided I wanted to run an IRC server for myself and a few of my friends. After looking at the availble options I settled on ngircd because it claimed to be simple to build and run, and to be highly portable. Remember, this is way before Joyent had implemented epoll on SmartOS.
I have a git repo that contains both upstream branches and patched branches on github. Feel free to spin up a 2014Q2 zone and follow along:
pkgin -y in build-essential libevent pkg-config irssi
git clone git://github.com/nahamu/ngircd
cd ngircd
Stock ngircd without epoll
Let's pretend it's 2012 and SmartOS doesn't have epoll yet (though for simplicity and to keep comparisons fair, we'll use the latest ngircd for all the tests.) We'll grab the stock 21.1 git tag and build it (note that I have to force it not to use epoll because otherwise it will automatically find the headers and use it...)
git checkout rel-21.1
./autogen.sh
./configure --without-epoll
make
The performance problem with this implementation will be immediately obvious once the first user connects. In three separate windows, run:
# launch ngircd
./src/ngircd/ngircd -n -f <(echo -e '[GLOBAL]\n Name = test.example.com')
# connect with irssi
irssi -c localhost
# launch prstat and watch ngircd sit at the top of the list chewing on CPU
prstat
On my machine ngircd was using 12% CPU (my server which has 8 logical CPUs so it's basically maxing out one CPU.) As an exercise for the reader, feel free to use truss or dtrace (or the source) to see what ngircd is doing. Whatever it is, it's not pretty.
modified ngircd using libevent
Now let's look at the performance improvement I was able to obtain a couple years ago by gutting the existing backends and replacing them with a very simple one based on libevent (original patch against ngircd-19.2 rebased onto ngircd-21.1). (Note that the output of configure will now be a lie. My code replaced all backends with libevent but I didn't update the configure code enough to change its output.)
git clean -fdx
git checkout rel-21.1-libevent
./autogen.sh
./configure
make
Again, in three separate windows:
# launch ngircd
./src/ngircd/ngircd -n -f <(echo -e '[GLOBAL]\n Name = test.example.com')
# connect with irssi
irssi -c localhost
# launch prstat and watch ngircd behave nicely
prstat
On my machine irssi is now listed above ngircd, though both are reporting 0.0% CPU. Clearly using event ports via libevent was a good idea two years ago.
As an aside, upstream wasn't interested in my patch because they prefer not to have external dependencies, and I wasn't interested in learning how to use event ports directly when I already had such a short clean implementation that worked for me. I rebased it onto the latest release when I needed to, and ngircd's clean source tree meant that it wasn't particularly painful to maintain my fork.
stock ngircd using epoll
Fast forward to 2014 and an implementation of epoll on SmartOS. Let's go back to the stock version and see how it performs when it uses the epoll API
git clean -fdx
git checkout rel-21.1
./autogen.sh
./configure
make
./src/ngircd/ngircd -n -f <(echo -e '[GLOBAL]\n Name = test.example.com')
Once more, in three separate windows:
# launch ngircd
./src/ngircd/ngircd -n -f <(echo -e '[GLOBAL]\n Name = test.example.com')
# connect with irssi
irssi -c localhost
# launch prstat and watch ngircd behave nicely
prstat
On my machine ngircd is back above irssi again, but both still report 0.0% CPU. That's pretty good considering I no longer need to maintain my patch. As long as I'm running on a modern version of SmartOS I can run stock ngircd and get good performance thanks to the implementation of the epoll API.
It's possible that I was getting superior performance from event ports, so if I ever feel like sharpening my C skills, perhaps writing a patch for upstream that can use event ports could be interesting.
For now, though, I can abandon my forked repo and go back to being lazy.
Closing thoughts
The very next commit after epoll was added to illumos-joyent was one that modified the lx brand to use it.
The "lx" brand dates back to the OpenSolaris days. It allowed you to run a zone that presented Linux APIs. You could run unmodified Linux binaries in a zone. (Okay, it pretty much had to be CentOS/RHEL 3.0 32bit, but still, unmodified!) At some point that work was abandoned, and it's no longer in illumos.
Those who have been watching the commits flowing into illumos-joyent are aware that engineers at Joyent have not only resurrected the lx brand, but have been working on modernizing it to run the user-space code from much more recent Linux distributions.
I'm fairly certain that the decision to implement the epoll API was based on the need for it for the lx brand work. And certainly, if the lx brand work pays off, very exciting things will be possible on SmartOS.
But even if it doesn't, the decision to implement these APIs and expose them in the joyent brand as well means that even if we can't run unmodified Linux binaries, if enough critical APIs are exposed, at the very least we should be able to compile lots of pieces of software that expect Linux APIs without having to patch them to use illumos APIs.