Technical Note 6

Protocols Within Protocols

I have found in my career working with the Web that complexity tends to snowball in what I'm inclined to call flat systems: all the logic (menial stuff like if the user is logged in, do this, otherwise do that) is piled up on top of itself because there's no depth—no space along a particular axis in which to spread out. MVC frameworks of the mid-2000s onward (such as Rails, Django…) add a little bit of depth but are still pretty flat. The newest stuff (React, etc.) has a teeny bit more of the kind of depth I'm talking about, but not a significant amount.

The axis in question is ultimately the time axis—the process of a single HTTP transaction:

The user agent issues a request,
the server procures✱ some content,
the server sends the content in response.

Flat systems bunch all their code up in the server procures some content phase, and I contend that the following are overlooked:

Manipulating the request: the URL, the headers, the request body (when present), and even the method are all available for twiddling. Manipulations here can sharply constrain the space of possible inputs, meaning downstream code can be much simpler. Access control, for instance, can happen here without mixing it up with content handling. URL rewriting makes it possible to dedicate microservices to specific functionality (as in, you put that functionality at specific addresses and rewrite requests to them), as well as provide the all-important bulwark against link rot.
Manipulating the response: everything that comes out of a content handler is (nominally) a string of bytes with an associated media type, and thus can be manipulated as such. This includes performing surgery on pages (changing the markup, adding metadata, rewriting links) and operations on images (crop, resize, etc.) and other media (extract frames from video, cut clips, etc.) Response headers and status codes are available for manipulation as well.

The challenge was to come up with a unified interface for both procuring content and manipulating requests and responses (which we can further generalize to manipulating HTTP messages), and while I've had the general strategy in mind for years, a couple weeks before the Summer of Protocols retreat, I managed to think out the details. It goes like this. Everything in Intertwingler is a Handler:

content handlers are handlers,
transforms are handlers,
even the engine itself is a handler.

A handler takes a(n HTTP) request as input and returns a response. A handler must respond to at least one address. All information must pass through that interface—no side channels, no spooky action, no exceptions. The purpose of architecting the system this way is threefold:

Extremely well-defined development targets. As a (back-end) developer, your work in the Intertwingler ecosystem is defined exclusively in terms of writing handlers.
A handler is effectively a stand-alone microservice, meaning it can be bench-tested in isolation, or even run in production that way.
This also means that handlers can be run on different systems or written in different programming languages, espousing the principle of intelligent heterogeneity.

At the time of this writing I'm not sure what kind of awful side effects this constraint is going to elicit, because I haven't written it all up yet, and reality has a surprising amount of detail. I am basing this design, however, on my experience writing Apache modules, where everything there is a handler too. It turns out there is a heck of a lot that can be framed as a handler if you get creative with your interpretation of the HTTP protocol, and there's a lot you can do in the respective departments of manipulate the request/response.

My hope is that eventually a library of handlers will grow up in the Intertwingler ecosystem, but I intend to provide a starter kit.

Pipes with Types

A transformation function, or transform, is a special case of handler—or more accurately, it's a special kind of resource within a handler. A transform is a URL that only accepts POST requests with predefined query parameters. Imagine the URL /transform/crop?x=50&y=50&width=100&height=100 accepts POST bodies containing images and returns the resulting image, cropped to the given parameters. When you GET /my/image;crop=50,50,100,100, the Intertwingler engine under the hood will locate /my/image and pipe it through /transform/crop—while remapping the path parameters into query parameters—before returning it, optionally caching the result along the way.

The above example is of a response transform which is directly addressable. Most transforms will be configured statically. Request transforms in particular don't make sense to make addressable (I ruminated on this for a while). There are likewise a bunch of stock response transforms (e.g. tack on social media metadata) which we will always want to run. This means a sort of sandwich-like structure for response transforms:

Statically-configured early-run transforms,
addressable transforms,
statically-configured late-run transforms.

That is, we want to always run certain transforms before and after the addressable ones. We also want it to be possible for an earlier transform, upon successful completion, to trigger the dynamic injection of another transform later on. It will be vital, for instance, for a request transform, after manipulating the request, to insert a partner response transform into the queue.

Transforms rely on the fact that a message body is just a string of bytes, and can be manipulated as such. An entire HTTP message also reduces to a string of bytes, and can be sent to a transform that returns a modified message with the media type message/http, which you then merge with your original message and use the result downstream. This works for both requests and responses and will in particular be essential for the viability of request transforms: you POST a serialized request to the transform and it gives you back a new one, and you just parse the result and merge it with the request you passed in. We can think of two subspecies of transforms, then: those that just manipulate HTTP message bodies, and those that manipulate the headers, or even the entire request. These can be differentiated easily enough in Intertwingler in the time being by stating that transforms that only deal in message/http or message/global-headers are entire-message-manipulating, while everything else only manipulates bodies. If that raises conflicts, we can explore other metadata options.

I will undoubtedly have to experiment with what kinds of configurations are viable, but my thinking is that request transforms are mainly going to deal with manipulating headers, or otherwise the entire request. I can think of one very useful request transform in particular that would only transform the request body, but the bulk of request transforms are undoubtedly going to mess with everything but the body (because most requests are GET which don't have a body).

I got this inspiration working on the forget-passwords project, which uses a lesser-known capability of FastCGI. It's a very clumsy thing where the server puts the script into an AUTHORIZER mode, where a successful (i.e., 200) response would indicate that the top-level request could proceed. You could also smuggle data out of this operation using specially-prefixed headers that get transformed into environment variables. Now, FastCGI is apparently out of vogue (ostensibly largely replaced by stand-alone services and beefy reverse proxies), and this was a bananas approach to the problem in question, so I asked myself why couldn't you just barf an HTTP message (request, response, doesn't matter) at a script and have it return a modified version which you then parse and use for subsequent operations? That is, why screw around with the weird vagaries of an ad-hoc system that the vanguard apparently doesn't like anymore, especially when a completely standard-compliant approach would do a better job.

What It Looks Like In Concert

This process invokes a request transform and three response transforms. These would all be statically configured, though the first of the response transforms should be bound to the early queue while the last two should be bound late.

The user agent issues a request to GET /some/page.
A request transform unconditionally adds text/markdown to the Accept: header—that is, if it isn't already there—and inserts its companion response transform into the queue.
The content handler locates a Markdown variant at the requested URL and serves it.
The companion response transform turns the Markdown body into HTML.
A subsequent response transform takes that HTML and tacks on all the social media metadata.
a final response transform rewrites all the URL references—links and such—to their best current values.

This process highlights the need for some mechanism for a request transform to hook a response transform, and moreover that the latter be inserted into the correct queue. Since transforms are conditional on content type, the transformation queues are only a partial order. it will be necessary

The Handler Manifest

Every handler will have a manifest. The manifest advertises what URIs the handler responds to, and what request methods are allowed on each URI, as well as what parameters are recognized, and what content types are expected and returned.

the manifest affords faster routing and autoconfiguration

every handler responds to OPTIONS * with its manifest

trailing slash paths are treated as prefixes