And Now I Promptly Digress

The single most useful source on the subject of computing I have ever read is a short book by Danny Hillis called The Pattern on the Stone, the first edition of which he published in 1998. Through this book he furnished me with the back half of a conceptual model for what computers are genuinely good at.

• A process—this is my definition—is any sequence of events linked together by a chain of causation. It also helps if it has boundaries in space in time, so you can describe it, analyze it, and compare it to other processes.
• A procedure is just a process that has been described with enough detail to be repeatable. Procedures also tend to have bills of materials and equipment attached to them, like a recipe. Follow the recipe, and, barring exogenous interference, you should be able to get more or less the same result every time. Furthermore, if you can write down a procedure really, really precisely, you can execute it by machine.
• An algorithm—this is Hillis now—is a procedure that implements a mathematical function. Now, in mathematics, a function is not a procedure—rather it is a relation that maps every element in one set to exactly one counterpart in another set. But if you don't know the counterpart, you have to find it out somehow. Computation—which precedes computers by millennia—means applying a chosen algorithm—which likewise predate computers by millennia—to a known element (the domain), to get its counterpart (range). Algorithms are amenable to formal analysis, which is useful for accurately gauging their resource consumption, but more importantly, an algorithm is guaranteed to always produce the same result for a given input.
• A heuristic is technically a kind of algorithm, but we can think of it as one that answers a different question than the one that was asked. Informally, a heuristic is a rule of thumb that helps you quickly make decisions when you don't have all the relevant information, with the trade-off that you might sometimes be wrong. This is the same for formal heuristics. Many practical problems are so costly to solve exhaustively that we opt for speed over the algorithmic guarantee. What this means is that not only can we not know if the answer is optimal, we may get different answers from one run to the next.
• Machine learning—Hillis said neural nets which, strictly speaking, is only one species thereof, but anyway this is me talking now—is what we now mean when we say AI. Machine learning can be understood as a kind of heuristic, where one of the inputs is the thing you want to ask about, and another is a big honking matrix of numbers. The matrix represents a distillation of a (potentially enormous) number of examples known as training data. The function answers a question like given what is known (represented by the matrix), what is the result most like the new thing (the other input)? —ultimately a linear algebra problem.

Framing machine learning as a special case of heuristic is useful, because of its inherently statistical nature. Inferences drawn from these systems will ineluctably sometimes be wrong. The perfect, deterministic equivalent would entail explicitly programming a case to handle every possible input, including input we had never seen before. As such, we trade off being wrong sometimes in return for being spared an incalculably huge effort in the setup. Every output of a machine-learning heuristic is therefore contingent on whatever happened to be in its training data. The kind of question any machine-learning heuristic poses is invariably something like, given what you've previously seen, what is the closest thing to X new input? This means that if what you've seen ever changes in any significant way, so will the ultimate answer.

Perhaps an Example is in Order

Old-school machine-learning systems are employed to turn fluffy things into crisp things—Boubas into Kikis. The caveat, should you choose to acknowledge it, was that the system might occasionally pick the wrong Kiki for a given Bouba.

How to think about this situation is how a bank might process handwritten cheques—an archaic system for transferring money that I want to underscore I believe is nevertheless perfectly ordinary and fine—and will no doubt be with us until money itself is somehow abolished. A cheque has a handful of free-form handwritten text fields for which it sure as heck would be useful to automate the process of reading.

For starters, a cheque already has a bunch of machine-readable data relating to its origin that has been naïvely OCR-able for decades. Second, a cheque requires the issuer to write the amount twice: first as a number, corroborated by a convention that spells out the same amount in text. Finally, the recipient written on the cheque will in most (but not all!) cases be the holder of the bank account in which it is to be deposited, and has to input the amount, providing two important points of context.

The contours, of how a system like this can fail—assuming there's enough money in the account—are on the order of:

• One or more fields is garbage, can't make any sense of it at all,
• Can't reconcile numeric representation of amount with textual,
• Can't reconcile recipient identity with depositor account,
• Can't corroborate issuer signature with that which is on record.

The remedy for any of these failure modes is always to kick the problem to a human delegate who will make their own judgment. It is clear that what AI does in this case is to vastly boost the number of cheques that can be handled without the intervention of a human overseer. It takes the Bouba piece of paper with ink scribbles on it, and turns it into a structured, machine-actionable digital message which can be operated over with Kiki business logic. The most consequential failure mode—that both the text interpreter and the numeric interpreter converge on the same value that happens to be wrong, and the cheque gets processed automatically for the wrong amount—is vanishingly unlikely. Even if that does happen, it's still not the end of the world. At least for the bank.

This is exactly the kind of AI that is genuinely useful, and it gets asymptotically better with more training data and bigger models to represent it. Even still, there's a point past which this extra effort (the training) and its outcome (the model) isn't worth it, and that point still very likely fits and runs comfortably on an ordinary commodity laptop. Not so for ChatGPT.

What Would Have To Happen In Order For That To Happen

Back to lawyers embarrassing themselves. Go through the back catalogue of Lawfare podcasts from the last several months and pick any one of them with AI in the title. Prepare yourself to be treated with a deluge of breathless, palpitating misunderstandings of what the technology is, what it's capable of, what direction it's headed, and how fast. Hours and hours of content have been minted by highly-educated, prestigiously-credentialed people, consternating about the policy implications of Sam Altman's speculative fan fiction, without stopping to consider the events that would have to occur first, for any of it to come to pass.

The narrative that artificial intelligence is rapidly accelerating toward AGI that will eventually outwit humanity's efforts to contain it, has gone unchecked by one important segment of the population: the people who write the laws, and the people who whisper into the ears of those people. What they're whispering is stuff like P(Doom): your personal confidence level (usually rendered as a percentage) that a rogue artificial intelligence—and not anything else—will annihilate humanity. A lot of things have to happen first for this to even be a possibility, let alone something you can assign a probability to.

Since it was Altman's spiel that got us here, let's pick on ChatGPT (or rather the underlying GPT-4), which is also easily the most famous. First of all, the P in GPT stands for pre-trained. Training took at least three solid months of compute over 25,000 or so repurposed high-end graphics cards, using the entire internet and all published text as input, and reportedly cost a hundred million dollars. The product of this training was a matrix (actually several matrices) containing a total of 1.7 trillion numbers, weighing in—presumably—at around seven terabytes. In order to function at all, these seven terabytes (or at least most thereof, as I understand it) have to be available in RAM at all times. Since the graphics cards only (only!) have 80 gigabytes of RAM apiece, they have to be yoked together—128 of them to be precise—with an ultra-high-speed network. Such is the GPT-4 inferencing rig:

• Four standard server racks,
• each containing four modules of eight NVidia A100 cards apiece,
• which, put together, costs over three million dollars,
• and draws about a hundred kilowatts of electricity,
• which is about as much as a typical suburban cul-de-sac at dinnertime.

Because of the RAM situation, even if you were the only user of ChatGPT, you would still need this setup to run it.

So, for starters, in order to fit something as powerful as ChatGPT onto ordinary hardware you could buy in a store, you would need to see at least three more orders of magnitude in the density of RAM chips—leaving completely aside for now the necessary vector compute. Call it ten Moore doublings, which, if everything stays on schedule, should happen sometime around 2040. Why I mention this is not for the purpose of imagining running one of these models at home per se, but has to do with a thing called the context window. This is like a session state that the model operates over, that acts like its memory: erase the context window and the model starts over fresh. The GPT-4 context window is 32,000 tokens, comparable to a feature-length New Yorker article. When you add stuff onto the end of a full context window, the older stuff at the front falls off into oblivion.

Why I bring up the context window is because all these fantasies of crafting bioweapons, or writing malware that conscripts military drones or launches nukes, or just plain tricking humans into letting it out in the first place, depend on a scenario of runaway self-improvement. A model—that seven-terabyte blob of matrices that costs three solid months and a hundred million dollars to mint—does not self-improve at this stage, because it is read-only. I submit that in order for a genuinely self-improving AI model, you'd need a context window at least as big as the model itself, because how can it self-improve if it continually forgets what it was doing? Saying the model needs an arbitrarily-large context window is the same thing as everybody having their very own three-million-dollar, four-rack vector processor array with ten terabytes of RAM to muck with—to say nothing of the 199 more identical rigs lined up next to it to do the actual training.

I have written before that I am actually sympathetic in principle to the idea that cognition more or less reduces to analogy at scale, and that brains (or I should say nervous systems at large, but neither are strictly necessary) are a particular type of analogy machine: One neuron fires into some number of other neurons, which, through some hodgepodge of internal mechanisms, determine whether to fire into some subset of neurons to which each is connected downstream. Lather, rinse, repeat for a structure that will take in signals, match them up to a set of stored signals it has already perceived, and respond accordingly. Firing a signal downstream is equivalent, at this instant, to forwarding one bit—as in binary digit—of representational state. If you skate over how that bit gets chosen and instead use a numerical weight, you get a mathematical structure called a weighted directed graph, which exhibits a statistical behaviour when you compute over it. Every directed graph, furthermore, is isomorphic to a thing called an adjacency matrix, which is fantastic, because that means you can do linear algebra to it (and quickly, with expensive repurposed graphics cards), and get some pretty impressive results.

What the AI proponents say, then, is that all we need to do is make a big enough matrix and we'll get something as smart as a human, and if we make an even bigger matrix, we'll get something even smarter than a human. Maybe? But we're talking about a matrix that's 100 billion numbers a side, which, even if it used half-precision (16-bit) numbers, would mean a 40-zettabyte matrix. In RAM. Luckily for these proponents, though, it would be mostly zeroes, because there are only (only!) on the order of 10,000 or so synapses between neurons. So you could probably compress that significantly, both for the purposes of storage and computation.

Now, there are cheats that cut that exponent down significantly, especially if the matrix is sparse (full of zeroes) which these models often are (and you can apparently cheat even more by rounding small values down to zero to make even more zeroes), but the computational cost of operating over a matrix is always going to be superlinear relative to its size. What this means is that an AI model can be made bigger, sooner than any reasonable amount of money, hardware, or electricity could power it.

Moore's law to the rescue? I'm not actually sure about that. Moore's law is more of an industrial performance target than a fact about the universe, and its original formulation considered cost as an essential part of it. So-called economic Moore's law has been over for a while: the newest chips with the tiniest features are more expensive to produce than the ones that precede them. So I suspect to get to intrinsically self-improving AI with enough pseudo-synapses to match a human's, we'd first need to see entirely new technologies for computing substrate—optical processors, carbon nanotube memory—that kind of thing.

Oh, but what if we get the AI to design its own next-generation substrate? Here's the problem with that: generative AI in its current form is little more than a bullshit generator. Remember? It's filled with Reddit and 4chan threads and has the working memory of a New Yorker article. It doesn't understand anything; it doesn't think. It can't innovate. It can't produce anything that wasn't once dreamed up by a living, breathing human. Those AI systems that do drug discovery or aid in materials science? Those aren't even the same species of thing. They're much closer to our humble cheque-reader than ChatGPT: discriminative systems with a narrow mission trained by supervised learning. ChatGPT can't just spawn one of these up on a whim, any more than it can successfully concoct a one-line sed formula.

And this is my final point: AI has no initiative. It doesn't want anything. It sits and waits for prompts, responds, and then waits some more. It doesn't get ideas or motivations to do things by itself. Maybe some enterprising engineer might rig something up? After all, even a humble thermostat senses its environment and responds in kind, and that doesn't even need a computer, let alone AI. To approximate anything like initiative, the AI would need a sensorimotor system: sensors, or something sensor-like, to detect information from the outside, and motors (not necessarily literal motors) to take some action in response. Furthermore, it would have to have some kind of internal representation of what good is.

This need not be complicated: good to a thermostat is the state of its bimetallic coil unwinding enough to cause the little blob of mercury to roll off the electrical contacts and over to the other side of the ampoule that makes up the mercury switch. So an AI system, in addition to its sensorimotor infrastructure, could conceivably be outfitted with a rudimentary axiological subsystem consisting of goods to pursue and bads to avoid. In other words, a simple matter of programming.

They Just Want a Slave

That's next on the roadmap though, right? AI agents? Like book-my-family-vacation-for-me kind of thing. Something that knows everything about you so it can anticipate what you'd like, has access to everybody's calendar and whatever so it can coordinate the timing, can write your boss (or hey, your boss's AI agent) to request the time off, has access to your credit card so it can book the flights and hotel, etc. Something that will proactively work on new ways to surprise and delight you, just like a real human servant.

I think this is a fantasy. It's concocted by people rich enough to already enjoy human servants, assuming—probably correctly—that there are people out there of lesser means who want the same kind of access. My instinct is that a product like this would be extremely touchy, and that's assuming you could even get it to work.

First, the only way this thing is going to get to know you is through its sensory apparatus. If one of your kids has piano lessons, that's going to have to go into a calendar that it can see. Maybe it does the initial data entry on that too? The point is, somebody or something is going to have to keep on top of sharing everything with this thing, otherwise it's going to make mistakes that stem from missing some key piece of information. Like a person, its decisions are only as good as its inputs. Unlike a person, however, I don't anticipate one of these things to be as good at improvising or triangulating, just plain picking up on vibes, or, crucially, understanding that it needs to add a new information source. I'm willing to be proven wrong though.

Next, I expect it'll have a hell of a time navigating interfaces—whether it fakes up a user to click on buttons or jacks straight into an API. I think it'll have trouble determining if it got the result it was after. The (discriminative) purpose-specific AI designed for looking at X-rays and such, keys off the wrong information all the time. Even being after something suggests having goals, but a multi-step process like arranging a vacation for an entire family is going to entail creating a composite plan, with subsidiary goals, conditions, principles and/or standing orders, modeling of interdependencies, elimination of internal conflicts, status checks, and alternatives in case something goes wrong. In other words, the artificial intelligence is going to need an artificial imagination.

I don't think the big matrix is going to cut it for this. Plans are Kiki. Text and images are Bouba. Good old-fashioned discriminative machine learning takes a Bouba and produces a Kiki (with the acknowledged risk of potentially being the wrong Kiki). Large language models take a Bouba and return another Bouba. In other other words, an AI's ability to plan is probably going to need a whole new architecture that has a Kiki alongside the Bouba, and that doesn't exist yet. In lieu of some smart person designing one, we're back to the AI that radically self-improves, which means we're back to the AI with the arbitrarily large context window and/or mutable model (which we already blew past several paragraphs ago), which means we're back to needing new technology for computing hardware to keep this proposition from being improbably expensive. So why not just rent it from Sam Altman?

Okay, one more consideration and then I'm going to walk away from this. People (law professors, policy advisors, tank-thinkers) have their knickers in a twist about alignment, an icky neologism referring to the extent to which the AI is acting in your interests rather than its own. When you're renting one of these things from Sam Altman, or Mark Zuckerberg, or Satya Nadella, or Sundar Pichai, or Tim Cook, or Jeff Bezos, or Elon fucking Musk? What about alignment qua their interests? The question to ask isn't is the AI working for itself instead of working for me? but rather is it working for them?

What's to stop these AI agents—again, assuming you could even get one to work—behaving in ways that benefit the people selling them at your expense? Using their preferred airlines and hotels, their brand of batteries and toilet paper? What's preventing one of these things from using your credit card to buy up a bunch of overstock its owner wanted to get rid of and framing it like it did something thoughtful for you? Or, what's preventing the owner from taking your private behavioural data, orders of magnitude more intimate than anything you've previously disclosed—voluntarily or otherwise—and packaging it up and selling it downstream? How would you ever know?

So, again, this is why I'm an AI meh-ptic. There are so many mundane social problems with the technology here and now that don't even remotely range into the territory of Skynet, Paperclip Maximizer/Grey Goo, Roko's Basilisk or The Matrix. So many huge, conspicuous, world-changing events have to happen before any of those sci-fi situations are even close to plausible, yet those are the things getting the policy attention. This is a serious misallocation of cognitive resources, and I urge those in influential positions to smarten up.