The problem that I'm trying to solve—or at least contribute in some small way to a solution—has been notoriously hard to articulate. I've narrowed it down to not one, but three points of departure:
If that proposition isn't valuable enough on its face, consider the problems early (and proto-) hypermedia pioneers were trying to solve:
All of these goals are contingent on making it super cheap to marshal lots and lots of tiny little pieces of information, connected to one another by an even bigger number of links—and this is important—links of all different kinds. Compared to its predecessors as a specimen of hypermedia, the Web, at least out of the box, ranks mediocre to poor. Without help, the Web is limited in its range of expressivity as a medium, because the cost of managing tiny pieces densely linked is prohibitive, and so it reverts back to the page being the basic unit of discourse. Again, nothing in principle prevents this from changing, so the question is, what could we make if the econophysics of the Web were dramatically altered?
And if that appeal isn't, well, appealing, consider that there is an entire class of tools that exist—or would exist—just beyond the reach of a spreadsheet. That is, they do very little besides afford data entry into some intricate structure or other, do some trivial manipulations, and then represent that information back to you—or perhaps somebody else. Many of the needs for these tools come from professionals in various niches—often very close the places such tools would be produced. The overhead, however, is still so high, that they need to wait for a vendor to come along and make an app for that
.
Personal knowledge management tools are beginning to fill this gap, but in my opinion they fail irretrievably in one important way or another. One such failure mode is platforms: the integrity of your content depends on paying regular tribute to some company or other (Notion, Roam) in perpetuity. In systems where you own your data
(Obsidian, Logseq), its disposition is ad-hoc. In either case, you lose a piece of what makes the Web what it is: a set of open standards and protocols that provide a single common interface, that doesn't have to be tailored to a single vendor, no matter how progressive they turn out to be.
The idea behind this initiative is to set a new floor, where the little guy has some bargaining power, not only because they own their data, but that data doesn't favour any particular vendor. This is about addressing a bottleneck to getting important work done as much as it is about addressing a power disparity.
The Web left a set of powerful capabilities on the table. Let's put them back.
The Web, out of the box, is a system for creating what I characterize as sparse hypermedia: long documents—pages—with few links connecting them. If you were to take a typical website and strip off the navigation bar and footer, the result may as well be a collection of Word documents or PDFs—in other words, print-era documents. In my opinion, one of the main value propositions of hypermedia is being able to spare people the need to read (or watch, or listen) any more than they have to. You accomplish that by breaking up the content into little pieces and making them subject to activation by the reader. I call this, by contrast, dense hypermedia. Dense hypermedia is what we had before the Web, from Hypercard and StorySpace, all the way back to the experimental systems and the original visions of hypermedia pioneers. What these systems had in common, though, is that they were all monoliths: hermetically sealed environments. What the Web did was make hypermedia permeable, able to cross administrative boundaries with no more effort than it takes to access a local resource. It is also characteristically easy to deploy: with the heavy lifting hived off to the poles of browser and server, the part in the middle—instantaneous global publication—can be had practically for free. The Web did these unique and powerful things at the expense of capabilities present in its predecessors, as well as introducing some problems of its own:
safelinks or state-modifying ones, such as form submissions), and you have what I characterize as
naïve embeds—basically just a rectangle of remote content plunked down wherever you put it. In addition to proper (seamless) transclusion, earlier hypermedia systems had stretchtext (telescopically-expanding detail), conditional display (e.g., Choose Your Own Adventure), and view control (different representations of the same content), among other interesting and powerful specimens of rhetorical, didactic, and literary value left on the table.
Again, the Web can do all these things in principle (and ultimately does accomplish many of them in practice), but the solutions are ad-hoc, one-off, and mutually incompatible. We're looking for a principled approach to restoring these lost capabilities, so they span organizations and implementations, as the Web does with its off-the-shelf repertoire.
After decades of experience with this medium, I conclude that much of what ails the Web can be traced back to the inherent tensions around URLs. HTTP URLs were initially derived from file paths. File systems themselves come from an era when storage was scarce and computers were typically not networked. As such the naming and locating of computer files—a necessary precursor to saving anything at all—wasn't something one did very frequently. In any case, what you chose to name a given file, and where you located it, was typically nobody else's business.
The cardinal sin of the Web was to attach public consequences to what had until then been a largely private affair. The state of the art is that URLs are shamefully unreliable, because there's nothing making them be. Link rot (and its subtler relative, content drift) is so bad, the median survival time of a URL is something like three months. Decades into this technology and there's still thin, if any meaningful, systematic support for the continuity of Web resources and the URLs that identify and locate them.
The tensions I alluded to have to do with diverging interests around the content of the URL string beyond its bare technical function. For one, URLs are important user interface elements: they lend themselves to being made intelligible, memorable, even guessable and/or inferrable. They telegraph information about the topological structure of the website, and they provide an entry point to the Web from any other (typo)graphical medium. URLs are therefore potentially valuable symbolic real estate, which gets even more valuable as they diffuse out into the environment. The tension, precisely, is that choosing a good, intelligible, future-proof URL takes considerable editorial and curatorial attention. At the same time, there is demand to pick one quickly, since you can't refer to (or even save!) the document until it has a URL.
The elephant in the room for URL continuity is other people's websites. The solution in principle is obvious: you can't trust other people's websites so you will have to police every outgoing link your website makes to ensure it continues to represent the same thing it did when you linked it. This is an easy (if labour-intensive) engineering problem in theory but a hard design problem in practice, so hard I am officially declaring it out of scope for this project. I do, however, endeavour for this project to demonstrate a model citizen
on the Web—that is, one you can depend on its constituent URLs pointing to the same thing in perpetuity.✱
What could we make on top of a substrate of dense hypermedia?
The introduction of the spreadsheet contributed to the M&A boom of the 1980s, and eliminated an entire class of journeyman accountant. It achieved this by shrinking the problem of financial scenario-planning to a point. 45 years later, spreadsheets are used for everything. They are a veritable Swiss-army knife of computing; arguably the only form of general-purpose computing available to the non-specialist public.
In fact, to date, the spreadsheet remains the only viable fully self-serve programming environment available to the general public. The problem is that there is a gulf between what can be accomplished with a spreadsheet, and what merits an entire team to be spun up to deliver an entire product. The gap is sparsely flecked with solutions like R and Jupyter Notebook, but they still require learning how to code. Merely knowing how to code isn't enough though, because of the colossal step change in effort once you exit the spreadsheet capability envelope. Many such targets aren't viable unless you divert your energy into creating a product. This of course is another entire universe of effort on top of what it would take to make a tool just for yourself. What this means, in the first place, is that a lot of problems simply aren't going to get solved.
The most significant advancement in spreadsheets in decades was putting them on the Web. That was actually a genuinely good idea. Same with the implicit version control and collaborative editing capability that came along for the ride. That said, I suspect people are finally outgrowing the fundamental constraints that make a spreadsheet what it is:
notebookproducts),
The existence proof that people want more can be observed in the proliferation of general-purpose products from AirTable to Notion to Roam, to specialized products (bug trackers, CRMs, even tip calculators) too numerous to list. I submit that each one of these is nevertheless missing at least one piece of the (second) brain. What makes these products viable is that unlike a spreadsheet-shaped problem, individuals can't fashion their own solution—or if they can, their solution is cumbersome enough that they prefer the niche product.
Imagine spreadsheets didn't exist and you needed a vendor to show up and make a separate app for every trivial little thing you use a spreadsheet for. What I'm arguing is that this is in fact the situation, just a little bit farther up a gradient of, not complexity per se, but let's call it perhaps intricacy of structure. There simply isn't enough software development capacity for there to be An App For That™, for every that.
[DIAGRAM TBD]
Once I figure out how I want to draw this spectrum-lookin' thing:
The idea is to move the line of what can be achieved by an individual (and is furthermore worth doing) to the right (and even fill out the left a little bit).
I posit that if we had, at the core, a way of entering, representing and exchanging structured data—not to mention its global, instantaneous publication—plus a handful of sundry computations, we could dramatically extend the capabilities of an individual knowledge worker, before they had to touch anything that looked like conventional, imperative code. A number of the problems that aren't worth solving because they currently require a lot of time to program—if not spinning up an entire team and company and capital to tackle—will be within reach. This implies that the companies who currently occupy this zone will have to move up the effort gradient. Some will go the way of those journeyman accountants. Oh well.
Two decades ago, the media theorist Lev Manovich posited that the database—by which he meant navigable hypermedia environments—would be the transformative medium of the 21st century, just as film transformed the 20th. I submit that we feel the effects of databases on us, like victims—ultimately of governments and corporations—but we still can't afford our own. After 25 years immersed in this industry—and I showed up late in the game—it's still a serious project to do anything on a computer that is more sophisticated than a spreadsheet, and waiting around for a vendor to serve my niche, esoteric needs—especially without onerous strings attached—is a laughable proposition. So what would happen if the unit economics of the database—or more appropriately, the knowledge graph—were to get dramatically cheaper?
Who owns your attention?
there isn't enough software development capacity to do all the things, and the capacity that is present extracts a shitton of rent
i'm basically arguing that the hole between spreadsheet and software that genuinely merits a company to be built up around it is bad for society
gotta talk about the intersection of:
While the germ of this project goes all the way back to , I got a boost in when I read Bret Victor's sprawling essay on climate change. I was particularly moved by the chapter called Media for Understanding Situations, in which he argued for the use of models and data in interactive simulations to inform public policy discourse. It was enough of a lift for Victor to do all he did to make the argument; we shouldn't also expect him to figure out how one would pull it off at scale—although he did put a dent in it. What got me interested was that the technical problem of marshalling data and computational models and making them available is one I actually know something about.
The complicating factor here is that Bret Victor wrote his essay in , when it was still possible to believe that if you just gave people good information, they would make good decisions. That is, before the bullshit renaissance that was in full swing . I nevertheless believe the technical problem is still worth solving, just perhaps not for the exact original reason.
Climate change is obviously Bad® and We Should Do Something™ about it. The We
in this case is typically shorthand for citizens of democracies. Except our governments only do what rich people want, and the things rich people want are typically good for them and bad for everybody else. (Or, if it is any good for us, it's better for them somehow.)
As wealth inequality sharpens, the dynamics produce a decreasing headcount of individuals, each with an increasing quantity of resources at their disposal. Everybody else is distracted with their own affairs, many of which are economic in nature. Too busy putting out fires to do things like participate in civil society. Now, when people write on inequality they usually mean income inequality and remark about how unfair it is. I'm not particularly interested in questions of fairness; rather I view extreme wealth inequality, when there are eight billion people on the planet and counting, as hazardous to human civilization.
[LORENZ CURVE SIMULATION GOES HERE]
I don't love Lorenz curves because they represent perfect equality as not only normative but somehow achievable. Perfect equality in an economic system is an unstable equilibrium that disappears the instant agents start interacting. And even if it was somehow stable, perfect equality would amount to political gridlock. What you're going to see instead is something power-law-ish, and the question to ask is what is the value of the parameter? Represent that as a Gini coefficient and the higher number corresponds to fewer people in charge of more resources each, which you can interpret roughly as how much of other people's attention can one of these individuals divert.
My thesis is essentially that people lose resolution as the pool of resources under their control increases: as the pile gets bigger, they just round off at bigger orders of magnitude. When individuals are in control of vast quantities of resources, even the smartest, most well-meaning person will make what are effectively targeting errors. What we're more likely to get, though, are people who are not well-meaning, nor even especially intelligent. This happens irrespective of whether those resources are yours. The difference with private wealth is it's your money, so you're only accountable to yourself.
The reason why I focus on wealth and not income is because wealth represents how much of other people's attention you can divert at any instant. Income is at best a derivative of that. (If you want you can think of—net, of course—income as—in the limit—what you get for your attention and wealth as how much attention you can command.)
Strategies for getting stupendously wealthy, feel free to mix and match:
Software systems contain these ingredients in unprecedented abundance.
When you're poor you can't speculate cause you can't afford to lose anything, ever. When you're rich, you can make tons of risky bets with huge antes and way way bigger upsides relative to any conceivable downside, lose most of them, and still come out on top, because—and this is true for everybody but it takes being rich to really leverage it—your adventures only have to pay off on average. There's also a huge chunk you can just burn without any expectation of return. That means copious vanity projects and white elephants, lavish consumption, wild speculative bets, and unending rivalries with peers. Rocket to Mars, indoor ski hill, solid gold toilet, giant limestone pyramid, ultra super megayacht, whatever. Do you really want the attitude of the people setting the global agenda to be I'll be fine regardless of the outcome, so YOLO
?
the final remark i'll make before i close this section out is wealthy people get a lot of leverage, that only becomes more pronounced under conditions of extreme inequality
human civilization has laboured under much more extreme wealth inequality than we see today but that was before the anthropocene
pick your favourite emperor; they could command entire nations but couldn't put a scratch in the planet's ecosystem no matter how hard they tried
climate change is a problem of the modern era basically a side effect of us being really successful as a species, of business as usual
but also a story of people who knew and sold that existence to us anyway
if we want to survive as a civilization we have to evolve new ways of organizing and people who do well with the current way of organizing are always going to get in the way of that
The sharpness of the inequality curve is going to track with the topology of the economic network: sharper reflects one that is more concentrated and centralized, blunter is more peer-to-peer and distributed. A gentler wealth distribution curve means the people at the top can't divert as much of everybody else's attention for their dumb bullshit. To blunt the curve, you have to be able to cut these entities out of your economic interactions. You have to be able to deny them your attention. In Bruce Sterling's words, keep more of the money yourself
.
One evolution in my own thinking over the last several years was a Copernican inversion in how I conceptualize information, belief, rhetoric, and reason.
okay what exactly do i want to say about this
something like more information means more misinformation
most instances of misinformation are barely consequential; people absorb a loss that amounts to a minor inconvenience and carry on
corollary: you can believe a lot of wrong shit and it doesn't matter
ooda loop is not natural, it's something you have to train into
people aren't looking for facts to integrate into a model to make decisions; they're looking for stories that justify what they're already motivated to do
(or what they've already done)
okay but what if you actually wanted accurate information, how would you get it
trusted sources (including identifiable authors), receipts, full bibliography, tamper-resistant, expiry date
also wait to respond (unless the outcome is worse if you don't)
may need to pay for good information if misinformation is sufficiently consequential
will need some kind of pricing model for that, ie how much is appropriate
big thing though is people who want good information should be able to get it
hypothesis is they will outperform the ones who insist on bad information (albeit subject to certain conditions)
the whole thing about voters getting the governments they deserve
i'm not pretending i'm saving the world here
radically more powerful ways of authoring, consuming, verifying information
radically more powerful ways of communicating facts
radically more powerful ways of storytelling
(sorry, the information density of substack blows)
XXX THIS IS RIPPED FROM ABOVE SO REWORK IT LOL
The problem with these niche products is that they are sold as subscriptions, and so the intricate memory palaces we create with them are contingent on plugging some meter in perpetuity. It would be one thing if it was just a matter of money, but there are mounting qualitative concerns with SaaS platforms that are becoming apparent as we gain more experience with them. To hedge against platforms, we need to be able to possess our own data—but the data is worthless if it can't be used outside the particular platform that disgorged it.
The problem that I'm trying to solve—or at least contribute in some small way to a solution—has been notoriously hard to articulate. Some prior art I can draw on for part of an explanation is the independent researcher Bret Victor's 2015 climate change essay, because climate change is one possible point of departure.
The essay is an exhortation to people trained in various technical disciplines, about specific things they can do to address climate change. In it is a chapter with the somewhat obscure title, Media for Understanding Situations. Victor uses as his example the situation of public policy discourse where quantities are involved—and what public policy doesn't involve (at least monetary) quantities? He remarks that it's possible to use the same data to argue either for or against a given policy prescription, simply by tweaking some of the variables. The conventional op-ed format, Victor argues, simply doesn't have the bandwidth to communicate this nuance, because it has to set the dials somewhere. His solution: hand said dials over to the audience, so they can determine where, if at all, is the sweet spot of parameters that intersects plausible, acceptable, and effective.
This essay was written a year before what can charitably be described—to use a technical term—as a bullshit renaissance. Its rise highlights a closely related problem: it is manifestly no longer sufficient—assuming it ever was—to simply give people information, though accurate information has scarcely never been more necessary. To animate the facts, there needs to be a narrative, and not every audience either understands or is motivated by the same story. A superposition of stories—that hopefully all draw the same conclusion—has to exist all at once. The medium for understanding situations therefore has to understand not only the situation that needs to be communicated, but also the various overlapping constituencies to whom a given situation must be communicated, and what their—often diverging, if not outright conflicting—expectations are.
Climate change, moreover, is a prime example of a global, existential, wicked problem:
So it is with climate change: there are people who have demonstrated that they are perfectly content with the prospect of living on a cinder, as long as they're in charge of it.
support communicators to communicate accurate information persuasively
you don't need accurate information to communicate persuasively, but if you're going to claim accuracy you'll need to cite your sources
and those sources will have to be legitimate
the state of the art of scientific communication sucks
basic unit is the paper; only just starting to make data available
no publication conventions for data
replication is in the toilet
Scientists don't need publishers as much as they need a publishing function.
science journalists often misinterpret results
we can posit a certain category of tool
it isn't really a tool as much as a thin wrapper around data
really just a way to create a constellation of small, structured, highly-connected chunks of information, and navigate around in it.
plus a handful of ways to manipulate how said information is represented
plus a handful of other operations besides
the pulverized bits are addressable, meaning they can be referenced and reused
pkm (tools for thought, second brain) are in this category
add collaboration support and message queue/task scheduling and you net most groupware (which also means, modulo scaling, you net most social networks)
a big subcategory is niche tools for professionals