I am currently seven weeks into my Summer of Protocols project, which is roughly to create an engine, a piece of software that imagines the Web with certain holes patched and dials turned, such that its dynamics are much closer to the hypermedia systems that preceded it. I have come to call these dynamics dense hypermedia.

The present strategy is to overhaul a software package that I had initially started in 2018 called the Content Swiss Army Knife, or by its in-language identifier, Intertwingler. I had initially written this software as something of a breadboard and laboratory for simply what I termed good ideas about Web content. In practice it is (currently) a static website generator (like Jekyll or 11ty) that marries a directory full of files with a wad of metadata and spits out another directory of files you upload to a Web server. I use Intertwingler for my own website, my book project The Nature of Software, as well as my client extranets.

The core of Intertwingler is probably about 10,000 lines of code, and will maybe grow to 20k with all the bells and whistles. The problem I'm facing at the moment is how those lines of code are to be distributed. I have rearranged this package a couple times already and it's the rearranging that costs the bulk of the effort. If I'm to rearrange it again, I want to get it into a state that will be relatively stable for a while.

At the moment, the bulk of the weight of Intertwingler is in two files, one containing the central Context class, and the other in an embarrassingly sprawling Util module. Almost all of the contents of Util module has already been siphoned off into a URI Resolver class, and a GraphOps mix-in that augments the adapter to the graph database with essential inferencing capabilities. The little that remains is still looking for a permanent home. Early on in the development of this software, the Context became the default location for every operation that wasn't rooted in manipulating existing files on the disk, such as analytical functions and generating pages whole cloth. This has the side effect of generated pages needing special treatment versus the bulk of the content. Rearranging the system so that both sourced and generated pages can be treated uniformly will likely empty out the Context object of all but the most rudimentary role of yoking together other subsystems and passing them their configuration.

Architectural Principles

Here is a summary of the most salient principles that come to mind when considering the design and implementation of this project.

Durable Addressing / Links Are First-Class Citizens
Without durable URIs, you have no dense hypermedia. To ensure durable backlinks, make (ordinary, forward) links into first-class objects.
Nodes and Links Are Typed
Typed nodes (i.e., pages) are a common means of selecting a visual presentation, but typed links make it possible to derive a lot of the layout.
Standard Interfaces
It is foolish to expect that any information system will remain a monolith of a single programming language or platform. Solution: make all the interfaces standard, to promote intelligent heterogeneity.
Layered System: Pipes with Types
Many operations in a resource-oriented information system (of which the Web is one) can be modeled as compositions of simple transformations over simple content. Many resources likewise can be modeled as derivations of other resources by way of some function or other. (This is absolutely cribbed from Roy Fielding.)
Wholeness
Every piece of a larger whole is a whole thing unto itself. (This is cribbed from Christopher Alexander.)

Revisiting Behaviour Sheets

The original reason why I started writing this note has to do with an exercise I was doing to help plan the reorganization of Intertwingler. It's a technique I came up with around I called behaviour sheets. A behaviour sheet can be understood as the penultimate description of an informatio nsystem; the most detailed you can get short of writing code. The way you write a behaviour sheet is as follows:

  1. Get an outliner, any✱ outliner will do.
  2. Create a heading for the particular software artifact (library, program, whatever) you're planning to write.
  3. Begin writing bullet points consisting of specific things the code must or must not do.
  4. Keep going until you've exhaustively described the behaviour the artifact.

Indentation in the outliner can be interpreted as something analogous to condition blocks. You will frequently discover that when you write down thing must do X, soon you (or somebody on your team) will ask but what about if Y happens? Capture that. Indent it under the initial statement. Then indented under that put [When Y], thing must do Z, along with whatever else it has to do.

I came up with behaviour sheets back in because I was trying to perform accurate time estimates on a rather tricky software development project. And they do yield extremely accurate time estimates, just with a catch: it takes as long to write a behaviour sheet as it does to write the code the behaviour sheet describes. If I start a behaviour sheet on Monday morning, I can tell you by lunchtime on Wednesday that the code will be finished by close of business on Friday. In other words, useless for forecasting—but, still useful for organizing work. It's not like the code would get written any sooner in the absence of doing a behaviour sheet, because you would have to puzzle out the same questions using a finer-grained medium.

Two remarks about behaviour sheets. The first is that they are not requirements in the conventional sense (even though strictly speaking they are requirements), nor are they features. You can describe features in terms of behaviour but you can't describe behaviour in terms of features. What I mean here is a feature in the software maps pretty much 1:1 to a discrete capability for the user, to the extent that a feature is the thing advertised in the marketing copy as what lets you do something or other. In that sense the existence of a feature is empirically testable: if the user can do the thing, then the feature must exist. This makes it an attractive (and stubbornly durable) entity for project managemment and marketing alike.

The problem with features, or at least descriptions of features, is that they are silent on details, and this makes them wide open to interpretation. If the user can complete the task then the feature is considered implemented and the requirement is considered satisfied. There is no accounting for side effects, or details of the process of carrying out the task.

And I say task here on purpose, as features are implementations of tasks, whether they complement the user in carrying out their task, or the computer carries out the task on its own. A task, however, is not a goal, and it's the goal that the user is interested in. A task is just one candidate in an entire universe of possible procedures for achieving a given goal. Indeed, there may be more than one way to do it, so overprescribing the method is a form of micromanagement that ultimately gets in the way of the user achieving their goal. Ditto risks, side effects and messes that the user has to clean up after carrying out the task. These concerns cannot be addressed in terms of features; they have to be addressed in terms of behaviour.

My other remark is about precedent. For starters, look at bug reports. They are typically written in terms of behaviour: thing does X when it should do Y, that kind of language. We might ask why the code has to be written and deployed first before this happens. What's really interesting though is how well behaviour sheets (and bug tracking habits) fit with earlier attempts at systematizing the process of developing design rationale.

Not too long after I came up with behaviour sheets, I did two salient things: I read Christopher Alexander's Notes on the Synthesis of Form, and started researching the work of Horst Rittel (which I gleaned by way of a remark Douglas Engelbart made in a video). The Alexander book (his 1964 PhD dissertation) is all about the topology of design problems, and how to find the mathematically optimal way to break hard problems down into a set of smaller, more tractable ones. Rittel, along with various colleagues, specified a framework for attacking wicked problems—a term he coined—called IBIS, or issue-based information system. This is effectively a knowledge graph containing three classes of entity:

Alexander's dissertation focuses on a concept of fitness variables which correspond to issues, and the bullet points of my own behaviour sheets were initially populated by something analogous to positions. Rittel's (et al) contribution, in addition to registering arguments, is the ability to model changes (through a generalizes/specializes relation) in conceptual scope. This is important because the decomposition algorithm Alexander proposed depends on the fitness variables (issues) being all the same conceptual scope. Behaviour sheets are likewise almost exclusively populated by the bottom concepts, but importantly these don't come out of nowhere. The rules asserted by behaviour sheets are necessarily informed by several layers of coarser-grained concerns. As such, I see IBIS as a viable platform upon which to develop a durable infrastructure that could serve as a backbone for an organization's planning efforts.

What I figured about behaviour sheets in the wake of my first experiments over a decade and a half ago was that while the product of a cold start was useless for forecasting, on account of it taking so long to populate, the standing concerns would eventually all get wrung out. What this means is that any new development would necessarily be informed by a mountain of prior art, that could easily date back many years, since these concerns don't change very quickly. In other words, the lead time to a viable forecast for any initiative would get shorter and shorter as the IBIS network grew.

What I'm envisioning, then, is an IBIS network (plus its empirical support materials) as an heirloom artifact within an organization. We could imagine it literally running the entire longitude of the organization's existence, accumulating incrementally as it ages. As parts of it grow obsolete, they can be archived—nominally kept around for historical purposes, but limited in their influence. At any given instant we can use Alexander's method—or rather, one of its contemporary descendants—to help intuit the structure, which can change dramatically as nodes are connected and disconnected from the graph. Behaviour sheets, then, just dissolve into the routine manipulation of an IBIS network at the more detailed end.

A reinvigorated IBIS tool is actually one of the first applications I plan to implement on top of the refactored Intertwingler. The prototype I made ten years ago is nevertheless in service on this project, but I find it easier to work with an outliner still, since the prototype (which was never intended to be load-bearing) is too sclerotic to use as-is as a daily driver, and fundamentally limited in a way that makes it more rational to replace than to try to fix. What I've done in the interim is put serial numbers (which double as cross-references) on each bullet on the outliner, prefixed I for issue, P for position, and A for argument. I put the behaviour sheet in the code repository so you can see what I mean. I also wrote a couple functions into Emacs to keep the numbering consistent so I don't have to manage that by hand. Once I have a viable candidate, I'll just write a little script that rips through the org-mode file and transforms it into a wad of IBIS data, which I can load—along with the other IBIS data—into the new tool.