On January 22, 2021, ten years to the day after installing it—a coincidence—I tried to get rid of OpenAFS. It did not go well.
Perhaps I'll preface: I am a staunch supporter of keeping residual computing and data storage resources on-premises. As onlookers have remarked over the years, I have operated my very own cloud
since around 1998, and I don't expect to change that policy anytime in the foreseeable future.
At the centre of this pattern of work is a server where all your stuff goes—almost always a Linux but there was some FreeBSD and even OpenBSD thrown in there earlier on—and then you connect to that nice, stable, sedentary place with your workstation, or laptop, or phone or whatever, whenever, wherever you are.
As with all things cloudy, the actual device in your hands is just a control surface for information somewhere else. Not only does this make them much less of a problem if they are lost or stolen, but it also makes all devices effectively interchangeable: I can, and have, in a pinch, borrowed a grandparent's Mac Performa to get some urgent work done.
Of course, wherever there are computers, there are files, and the question arises: how do you get files on and off these control-surface devices? Moreover, how to do it when the parts are heterogeneous? Indeed, at one job back in 2004, I had a Mac laptop, a Windows XP workstation, and a Debian 4U rack-mount server under my desk. The only viable answer, for schlepping files betwixt, was Samba.
Samba, of course, is the salsified, reverse-engineered Linux/Unix implementation of Windows file sharing, also known as Server Message Block. SMB is ubiquitous because Windows is ubiquitous: everything ships with it.
At some point on this fateful day, one of the disks on my RAID 1 died. No matter, I thought, and went and bought a matching drive to replace the dead one. As the RAID was rebuilding, it became clear, for reasons that are lost to time, that the file system itself was corrupt, resulting in the permanent loss of a random subset of files on a certain region of the disk, a few of which I didn't have backups for.
Heretofore, I had been following my regular pattern of using Debian (now Ubuntu) as my main workspace, and $WHATEVER as a front-end, which at the time meant my Mac Mini. Naturally, I used Samba to schlep files back and forth. What I found was, whenever there was even the slightest network interruption, the file share would drop. Worse, if I ever restarted the server while a share was connected, it would hang on the Mac, incapable of being rehabilitated short of a hold-down-the-power-button restart. I remember reading that AFS was specifically designed to work around service interruptions✱, and since I had all the floorboards torn up and I had a hankering to try Kerberos for some other stuff, I figured why not give AFS a shot.
Ten years later, I tried to get rid of it. Fun fact: I know it was exactly ten years because I caught a journal entry dated whizz by when I was refreshing my backup on , in preparation for the teardown.
If you want to run an AFS server, it comes in one flavour: OpenAFS. The genuine article, or one of its derivative works, is the only viable, up-to-date, production-ready implementation. It is a weird piece of software, that doesn't look like most server software you see on Linux, because it comes from IBM, before Linux was even really a thing.
In my decade as an (Open)AFS user, I have noticed some…concerns. I have divided my concerns into the AFS (P)rotocol in general and the OpenAFS (I)mplementation in particular.
/vicepa.localway to access files in OpenAFS, even when it is hosted on the same computer. You can only access it over a UDP socket. Bonus: you have to tune the packet size down if you want to access the cell over a VPN, or file transfers will mysteriously choke.
pagshpagsh to work.These accumulated annoyances led me to decide it was finally time to give up AFS for something more mainstream—something like Samba.
My recollection of Windows networking in general, and Samba in particular, is that it is a relatively Mickey-Mouse protocol whose implementation is nevertheless fairly easy to configure, and literally everything supports it. It has since grown up to become a robust and efficient networked file protocol with nice features like strong encryption and authentication, especially with Kerberos. Apple, furthermore, ditched their own AFP in favour of SMB✱, making it ostensibly the Common Internet File System
Microsoft originally hoped it would.
I figured Samba must have matured considerably in the last decade, given that they now have a drop-in replacement for Windows Active Directory—a snarl of protocols and databases that make corporate office networks Just Work™. I was further emboldened by Apple's move✱, figuring SMB couldn't be all that bad anymore. At any rate, the idea that I wouldn't have to go around configuring clients was very alluring.
I thus resolved to set myself up a Kerberos-authenticated Samba. The path of least resistance looked like spinning up some portion of its Active Directory functionality.
What wasn't clear at the outset was that the Samba Active Directory implementation kinda duct-tapes together its own Kerberos, LDAP, and DNS services. Well guess what: I already have Kerberos, LDAP, and DNS services running on my network. On the same machine, even. And I'm using them for stuff. Blowing them away for the sake of Samba's funky AD thingamajig is out of the question.
Now, this didn't answer the question for me if I could just connect to plain Samba via Kerberos, because it wasn't obvious in the documentation that I could. I didn't find out because I was surprised by something else…
As I was fiddling around trying to get Samba to work with Kerberos without having to haul in the Active Directory stuff, I figured I would try to get it working first without Kerberos, just because.
It turns out that Samba out of the box makes the Unix permissions on all the files show up wrong. Why? Because it's trying to map them to DOS permissions somehow. Well how in the hell are they going to service the growing non-Windows world if all they have on offer is DOS permissions?
Wait, okay, there is a configuration parameter unix extensions = yes and a few others that shove the DOS permissions into extended file system attributes, but that has no effect. I finally mount the share on Linux and determine that it's actually the Mac client that's not cooperating. Hmmmm…. After slogging through StackOverflow some, I trip over a wiki document and concomitant mailing list exchange that yields a number of obscure configuration parameters specific to satiating Apple SMB clients. I copy and paste those into the config, restart the service, and everything on my Mac appears to be fine.
Except it isn't fine.
None of my symlinks are showing up as symlinks; they're showing up as files. What the hell? I could have sworn Samba supported symlinks. How the heck did Apple handle this? They ditched Finder aliases for proper POSIX symlinks decades ago. There is no way they would have switched to SMB without solving this problem.
Well it turns out, Apple solved the symlink problem by ditching Samba altogether in , when the latter changed their licensing terms. That's how they solved it: they rolled their own. (I told you I'd get to this!)
What about those symlinks, anyway? Well it turns out, that what Samba means by symlink
is actually two distinct things. There are ordinary POSIX symlinks, and then there are these weird ersatz objects that show up as symlinks over the network, but on the local file system are plain text files✱. Real
symlinks are still represented to SMB clients as regular files and directories, meaning funny things will happen if you try to move them around.
What I want, is a faithful representation of the important subset of my server's file system, accessible to other machines on my network, including all permissions and symlinks. Version 3 of the SMB protocol itself, does support proper POSIX semantics, but the Samba team doesn't consider it a priority. There is experimental code, but you have to enable it explicitly at compile time, and I don't feel like maintaining that. Moreover, I never got my answer with respect to Kerberized SMB without the Active Directory dog-and-pony show, and I know it still disconnects at the slightest network hiccup.
I guess I never knew how good I had it with OpenAFS.
This quixotic excursion took me from the afternoon of , to the morning of the , while taking the weekend off. That was entirely too much effort to end up more or less back where I started—well, I guess I learned some stuff.
Most of the time was wrapped up in copying hundreds of gigabytes of files out of, and then back into AFS, poring over documentation, and interacting with the Samba mailing list. I even streamed some of the process, but as soon as I realized it would be even more boring than my usual transmissions, I promptly shut it off.
This story isn't quite over yet. Again, what I want is a POSIX-enough
file system, accessible over a network. Moreover, for the purpose of performance, backups, mmap()-able databases in project directories, etc., I want a properly-local file system, physically on the server, which is exported to the network. I want any manipulations to that file system by any network client to be faithfully represented on the server's local file system. I want it to be relatively robust to network outages—at least no worse than AFS is. Bonus if I can reuse my existing Kerberos setup.
As such, I have my eyes on NFS version 4, which looked initially like it only authenticated—even under Kerberos—at the host-to-host level, but it seems like you can actually authenticate with any principal you like. This is good, if the advertisements are accurate. It means that the infamous UID/GID mapping problem that typically makes NFS a non-starter, is taken care of by Kerberos.
If NFSv4 ends up working, I will once again say adieu to OpenAFS—you gloriously weird, misshapen product—and probably run Samba on the side, for my phone or whatever.
I don't know how you can really do this stuff without just diving into it. When I started this little adventure, sure I had a bunch of wrong assumptions, but it isn't clear that I would have dispelled those assumptions with any less effort than crashing headlong into them. Maybe if I had pored over the entirety of the Samba documentation before getting started? Maybe if I had read every one of the SMB specifications? I doubt it: documents like those are of the scanning variety, and you can glaze right over something important if you aren't specifically looking for it. But more to the point: it would have taken longer to read all that documentation cover to cover than just jumping in and trying stuff, and a good measure of it probably would have been wrong anyway.
Maybe I would have tipped myself off if I had thought to read up on Apple's departure from AFP—and subsequently Samba, and rolling their own SMB implementation? Even if I had tried to set up my Kerberized Samba before nuking AFS—which I was going to do anyway to introduce encryption on the underlying drive—it isn't clear that I would have caught the symlink issue in a testing environment, because it wouldn't have occurred to me to explicitly test it. In short, I would have ended up right where I did. It would have taken twenty hours any way you slice it.
Anyway, back to work.