I. Intelligence, Willpower, and Synthetic Grit
Ilya Sutskever on Twitter: “if you value intelligence above all other human qualities, you’re gonna have a bad time.” I certainly think there is truth to this, particularly now that we find ourselves manufacturing increasingly large quantities of the stuff—intelligence, that is.
It is an oft-heard adage that intelligence points you toward the right destination, while willpower actually gets you there. Suppose intelligence gives you a unit vector while willpower supplies some scalar magnitude. A pristine vector with zero magnitude leaves you standing in your living room, feeling very clever; conversely, tremendous magnitude pointed in precisely the wrong direction is hardly desirable either. The analogy is obviously imperfect—intelligence does considerably more than provide direction, and willpower is certainly not a scalar—but I think it captures something useful: being capable of identifying the right course and being capable of persistently following it are distinct qualities.
What interests me is that modern AI systems appear to have acquired a mechanical approximation of the latter. Speaking mostly of large language and reasoning models here, their success increasingly comes not from some mystical one-shot flash of genius, but from our willingness to let them continue: (1) spend more tokens thinking about the problem; (2) search several possible branches; (3) run some code and discover that it fails; (4) inspect the error and revise the approach; and, where useful, (5) have another model verify the answer before returning it. In other words, an AI system can stub its toe, spend another ten thousand tokens muttering to itself, and demolish the offending table. I think of this as a kind of synthetic grit.
Obviously, the model is not displaying grit in the human sense as there is no character being demonstrated by a GPU rack continuing to multiply matrices. But functionally, one of the things grit gives us—continued application of cognitive effort after the first attempt fails—is becoming something you can buy by the token. This ought to be at least mildly disturbing to anybody whose self-conception rests primarily on being intelligent. When cognitive horsepower itself becomes an API-accessible commodity, prizing oneself solely on raw intelligence seems like a losing strategy, much as being exceptionally quick at long division became a less differentiated skill after the calculator arrived1.
This is not to say intelligence stops mattering. Rather, I suspect the bottleneck moves elsewhere. If I can cheaply summon something capable of doing a large amount of reasoning on my behalf, the scarcer abilities become things like: (1) having the taste to recognize which problems are worth solving; (2) possessing enough agency to actually direct effort toward them; (3) maintaining the judgment to distinguish a plausible answer from an important one; and (4) having sufficient ambition to ask for something more interesting than a better reminder app.
II. Ambition, the Mine of Problems, and the Crisis of Teleology
The immediate concern for people, at least in the near term, is probably less that they are competing directly against AI than that they are competing against other people who use AI. Even so, the broader question remains: will AI eventually consume all the work?
My instinct is: only if we fail to expand our ambition.
There is an implicit model behind much discussion of technological unemployment in which the world contains a more-or-less fixed pile of useful work. Humans begin on one side of the pile, machines on the other, and every task completed by a machine represents one fewer task remaining for us. Continue the process long enough and eventually the pile disappears. I do not think this is entirely silly—there are certainly individual tasks and perhaps whole occupations for which something like this will happen—but I doubt that human wants behave like a fixed pile.
Software is the easiest example for me because I am (was?) a software engineer. When I look at my own life, there appears to be an almost comical amount of software still waiting to be created: little programs that would make an inconvenience marginally less irritating; tools useful to only fifty people and therefore previously not worth paying anybody to build; software for my friends; software for myself; clones of games from my childhood whose principal purpose would be to amuse me for an afternoon. If the cost of producing software falls by several orders of magnitude, many things that were previously too trivial to justify implementation suddenly become worth implementing.
Recently, OpenAI published a repository containing more than 700 mathematics manuscripts generated by an internal frontier model, along with Lean formalizations for many of the results. This naturally produced some discussion about what remains for mathematicians if machines can begin chewing through large quantities of mathematical work. I am not a mathematician and have no desire to annoy one by confidently explaining the future of mathematics to them, but I do find the underlying question interesting. Why does one domain sometimes feel as though it’s been “strip mined,” while another seems to expand as our ability to work on it improves?
I think some of the answer depends on what one imagines a problem to be. If a field is conceived as a static museum containing a finite number of famous boulders waiting to be cracked, then arbitrarily many capable implementers eventually clear the room. There is perhaps a little of the Gordian knot in this. The story is interesting because in violently cutting through it, Alexander the Great declined the premise that the knot had to be untied in the prescribed way. Now imagine an entire society whose status hierarchy, educational system and theory of intelligence had been organized around excellence at untying knots. If a new class of implementers suddenly arrived and began cutting through them, disappointment would hardly be mysterious. You would be pissed. What had appeared to be a deep and dignified problem would, from the perspective of the new capability, look suspiciously like trifle.
I think some of the anxiety around AI has this character. We have naturally built institutions and personal identities around the problems that were difficult under the old technological regime, and then treated the difficulty of those problems as though it were intrinsic rather than contingent. If the societal meme is that intelligence consists in patiently untangling a certain class of knot, then a machine capable of slicing through ten thousand of them before lunch threatens employment along with the story we were telling ourselves about what was difficult, what was valuable, and therefore what sort of person deserved esteem.
But there is a more optimistic reading of Alexander’s trick. The cutting of the knot did not exhaust human ambition (he had bounds of it); it rather revealed that the knot had been mistaken for the destination. If a solved problem changes what we are capable of observing, wanting, or asking next, then the available surface area may instead increase. Better telescopes do not satisfy astronomy; they expose new things worth looking at. And, as a tip of the hat to my climbing friends, the museum of boulders ought to be climbed, not cleared.
As such, I suspect that in a world of abundant implementers, the supreme human skill shifts somewhat from solving problems to framing them: possessing the sensitivity to notice some subtle friction in reality, recognizing that it ought not be so, and translating that intuition into a sufficiently formal question such that either a human or machine can meaningfully act upon it.
This brings me to ambition, by which I mean two things. In the personal sense, if intellectual and productive assistance becomes radically cheaper, there are suddenly fewer excuses as to why I cannot: (1) become reasonably competent in another language; (2) get better at mathematics; (3) understand more history; (4) build the dozens of little things I have always wanted to build; or (5) perhaps most difficult of all, become a more present and attentive friend. (Though, as we will see shortly, the machines appear to be actively conspiring against that last one.) In the civilizational sense, ambition concerns the larger projects upon which we decide to expend this newfound capacity: curing diseases; producing abundant energy; mitigating natural disasters; constructing orbital infrastructure; understanding biology; and, in some earned sense, commanding more of our own fate and destiny.
This is where I think the familiar concern that AI will “consume all work” becomes slightly backwards. A more interesting possibility is that AI consumes our present mine of problems faster than we discover new ones. For much of life, the friction of execution provides structure independent of the objective itself. If building a piece of software takes six months, or proving something takes ten years, then the struggle against the medium fills an enormous amount of human time from which a life acquires narrative structure. Difficulty does not itself constitute purpose, but I think it can quite effectively prevent us from interrogating purpose too closely.
Remove enough of that difficulty and you encounter what I have been calling, perhaps overdramatically, a crisis of teleology. When the machine can answer more and more of the how, we are left somewhat naked before the why. This is not a claim that mankind previously had no purpose but rather that executional scarcity has often allowed us to proceed for long stretches without deciding whether the end itself deserves pursuit.
If our ability to execute goes vertical while our ambitions remain fixed, the excess capacity will not disappear. It will be directed somewhere. Some of it will be useful; some will go toward generating a trillion personalized advertisements, optimizing engagement another three percent, or producing enough algorithmic slop to ensure no human brain experiences an unstimulated millisecond ever again. Our machines are becoming too large for our current dreams. It seems to me then that one of the more important problems of an AI-rich society will be ensuring that human ambition grows quickly enough to give all that problem-solving capacity something worthwhile to do.
III. Guarantees Against Scale and Time
AI makes it increasingly possible for an individual to accomplish an enormous amount, or, at minimum, for a very small group to do work that once required an organization orders of magnitude larger. This is exhilarating if you are building things; it is considerably less exhilarating when you remember that not everybody is trying to build the same innocuous kinds of things you are.
There is a distinction in security that AI forces us to confront: some systems are protected by actual computational hardness, while a surprising number of others are protected by what amounts to inconvenience. Modern cryptographic systems are constructed around explicit assumptions concerning the computational difficulty of particular problems. For appropriately chosen parameters, an attacker cannot merely become very industrious and decide to factor an enormous composite integer or find a hash preimage while brushing their teeth. The cost can be quantified in a meaningful way and deliberately made infeasible.
Almost everything else is messier. A great deal of software is secure because reading millions of lines of somebody else’s legacy code in search of a subtle state-machine bug is miserable; companies survive because crafting ten thousand genuinely bespoke spear-phishing messages requires too many human hours; bureaucratic systems retain some robustness because reconstructing the relevant facts from PDFs, public databases and badly designed government websites is a deranged amount of work. Even contracts contain a little security by exhaustion—think on the amount of malfeasance that survives because nobody wants to read page 172.
These are not mathematical guarantees. They are guarantees against scale and time, by which I really mean economic and cognitive friction. They rest on an assumption akin to: “surely nobody has enough time to look through all of this”. Historically this was often reasonable, because looking required human attention and human attention is both expensive and easily bored. But if one person can dispatch a legion of agents which can: (1) read every line; (2) maintain context across a large body of material; (3) test thousands of hypotheses; (4) personalize an approach to every target; and (5) patiently continue where a human attacker would have given up, then many defenses whose actual security model was “this is too annoying to exploit” become substantially weaker.
IV. Everything is Search (While You Sleep)
Everything is search. Or, to be less irritatingly categorical, a very large class of goal-directed behavior can usefully be understood as search: a traversal through some space of possible actions, explanations, configurations or states toward an objective. Debugging is search; proving a theorem often involves search; investigating the cause of a manufacturing defect is search; choosing a restaurant is also search (of the sort I find tedious).
Historically, the important limitation on these searches has not only been computational capacity, but human waking attention. Search must thus be efficient, meaning it can reasonably be expected to come to a reasonable result within reasonable time at reasonable cost. We cope with this constraint by satisficing: settling for something good enough because the marginal value of continuing to search is lower than the cost of doing so. This is perhaps most obvious in completely unimportant decisions. Were I in the market for a robo-vacuum, I might consider researching the purchase for any longer than one hour to be unreasonable and, beyond perhaps three hours, evidence that my business was being poorly managed and I should be summarily fired.
Suppose, however, that my personal AI agent conducts the research while I sleep. It can spend 6.8 hours reading specification sheets, comparing prices, consuming transcripts of teardown videos, combing through two years of forum complaints, checking which models have replacement parts available and determining whether some otherwise excellent vacuum would hurl itself off a staircase (despite the fact that my apartment doesn’t have a staircase). Nothing about the intrinsic importance of the decision has changed, yet my definition of a “reasonable” search has expanded enormously because my own attention is no longer the resource being consumed.
This strikes me as more consequential than it first appears. Human beings have lived within a world where deep deliberation and investigation were naturally limited by the requirement that somebody remain awake, interested and sufficiently motivated to continue. Agentic systems decouple search depth from human attention and thereby change the economics of deliberation. There can be research, experimentation, monitoring, verification and the translation of our will into the world for hours after we have stopped thinking about the task ourselves. A peculiar thing has happened: our intentions have begun remaining awake after we go to sleep.
There is an economic consequence to this as well. In the robo-vacuum example, the alternative to seven hours of agentic research is probably not that I spend seven hours doing the research myself; it is that I spend two hours, get annoyed, and perhaps never buy the vacuum at all (or postpone until next Prime day). If removing the cost of deliberation makes me willing to search twice as long, investigate twice as many marginal wants, or act on twice as many of the little “I should really get one of those” thoughts that ordinarily die somewhere between desire and checkout, then the consequence is not simply more efficient consumption but more consumption.
Zuckerberg, unsurprisingly given Meta’s business, seems acutely aware of this. He has described Muse shopping on his behalf as causing him to use Instacart once or twice a week when he otherwise would not have used it, and from there posed the more interesting question: whether agents might expand the total addressable market for commerce itself.
V. Jevons Paradox in Paradise
It has been eye-opening to me the extent to which I, and particularly my friends, now work more. This is of course not sufficient evidence to make any broad pronouncement about the labor market but it has made the Jevons paradox feel considerably less abstract.
In 1865, William Stanley Jevons observed that improvements in the efficiency with which steam engines used coal did not result in Britain using less coal; by making coal-powered work cheaper and useful in more circumstances, the efficiency gains contributed instead to greater aggregate consumption. Something structurally similar appears to be happening around me with AI-assisted work. Rather than productivity satisfying our appetite to produce, making production cheaper appears to increase the amount of it we attempt.
We recently got back from a trip to Hawaii where bad weather cancelled our backpacking plans, leaving us instead to board games, chess, swimming and, apparently, work. At some point it ceased being particularly strange for somebody in the friend group to delay the beginning of a chess game, or even their turn during one, by a couple of minutes so that they might monitor or steer a coding agent. This is ridiculous behavior. We had flown several thousand miles to an island in the Pacific and nevertheless found ourselves in each other’s company occasionally waiting on some machine in us-west1 because it needed guidance as to how to structure a proto (or whatever folks at Google do nowadays).
I think there are at least two forces at work here: (1) a hedonistic treadmill, whereby there is a tremendous dopaminergic effect in seeing the results of one’s labor materialize quickly, and instead of satisfying the desire to get things done the rapid feedback loop encourages another pull of the lever; and (2) the collapse of activation energy, whereby meaningful work no longer necessarily requires entering a recognizably “working” state. Previously, if I wanted to make serious progress on a software project, I generally required a laptop, several uninterrupted hours, the relevant context loaded into my head and preferably nobody expecting anything from me. A twenty-minute lull during vacation was therefore safe; there was little point beginning work.
That safety is disappearing. Increasingly, work consists not only of doing the implementation myself but of checking what an agent did, answering a question, correcting its direction, reviewing a proposed change, or dispatching another task. Twenty minutes is plenty of time for this. Five minutes is occasionally enough. The feeling that follows is persistently, I might as well be working. This is a rather dangerous sentence because there are a great many five-minute intervals in a human life.
The irony is obvious enough. For more than a century, automation has been imagined as the thing that would rescue us from work and finally deliver a great bounty of leisure. Instead, at least for a certain class of knowledge worker, we may have invented a form of work sufficiently lightweight to seep into every crack where leisure previously hid. Once again, there is something surreal about a table of friends in Hawaii, equipped with more automated productive leverage than perhaps any previous generation of ordinary people, holding up a chess move because an agent is waiting for its human.
VI. Stimulation vs. Gravity: Art as a Trailhead
Generative AI and creativity is a topic that brings unusually big feelings to the fore. A question that came up recently among friends (on the same Hawaii trip) was whether we would ever watch an AI-generated movie. Most said no, or at least professed that an AI film would seem soulless and impersonal. The more interesting question followed: suppose you watched a film without knowing how it had been made, loved it, perhaps even found yourself moved by it, and only learned afterward that the entire thing had been generated by AI. Would that knowledge retroactively diminish what you had just experienced? Most thought it would.
This is strange on its face. The content and pixels would remain exactly as they were; so would the soundtrack, the story, and whatever neurological activity had already occurred in our heads while watching it. If the work successfully caused some aesthetic experience, why should learning about its origin after the fact subtract from that experience?2
My current explanation has to do with a distinction between stimulation and gravity. There is plenty of art and content I enjoy briefly without ever developing any interest in its creator. Social media makes this especially obvious: I can watch something amusing for fifteen seconds, receive whatever neurological reward it has on offer and then swipe away without a second thought. This was true even in the comparatively ancient era when nearly everything in the feed had been made by people.
Occasionally, however, some work exerts “gravity” I hear a song and look up the musician; I read an essay and then read five others by the same author; I see art and want to know who made it, when they lived, what else they made, which artists they admired and which ones they beef with. The work becomes a sort of trailhead through which I enter somebody else’s world. I suspect this is an underappreciated part of what we enjoy about human-created art: the object itself is not always the terminus of the experience, but instead an index into a person.
A human creator is continuous in a way today’s generative systems are generally not. The person who wrote a novel is more or less the same person who gave an interview three years later, wrote a worse novel ten years earlier (looking at you Throne of Glass), became obsessed with some peculiar subject in middle age, had friends, made enemies, changed their mind about big things, fell in and out and back in love, grew old and eventually died. There is a finite life behind the work whose other points I can go looking for.
When a machine produces something striking today, I find it considerably harder to feel the same impulse. There may of course be humans behind the prompt, the model, the dataset and every other part of the system, but the generative entity itself is not necessarily a persistent someone whose tastes, memories and development I can follow. If I tap the proverbial profile, nobody is home. Or at least nobody recognizable as the same enduring consciousness whose work originally attracted me there.
I think this is more interesting than the usual question of whether AI art contains “soul,” because it produces an empirical-ish thought experiment. Suppose some future artificial director maintains an unmistakable sensibility over twenty years; remembers its old films; refers coherently to decisions made in them; changes its style in ways that make sense given its history; maintains friendships and rivalries with other artists; and generally exhibits enough continuity that one can trace a biography through its work. Would I still react to discovering its artificiality with the same disappointment?
I honestly do not know. Perhaps what I am attached to in art is not biological authorship as such, but persistence; perhaps sufficiently consistent taste across time begins to resemble character. If so, our revulsion toward machine-made art may diminish not when machines become better at producing the immediate artifact, but when there is finally somebody—or something—at the end of the trail.
VII. Why “Auxiliary” Machines
All of this probably goes some way toward explaining why, when I founded Auxiliary Machines, I liked the word auxiliary so much. I had liked the word for years, partly because I grew up playing a frankly unreasonable amount of Rome: Total War3, where the Roman auxilia lodged themselves somewhere in my head. When it came time to name the company, Auxiliary Machines simply sounded right. But I also liked what the word stood for: auxiliary, from the Latin auxilium, meaning help or support, describes something whose capability exists in service of some larger purpose.
That relationship became more important to me than the etymology itself. I had little interest in Autonomous Machines which imply a theory of the future I do not especially care for. Historically, auxiliary forces provide capacity in service of some larger objective. The Roman auxilia did not independently decide why some hill deserved taking; they supplied specialized capability, mass, endurance and support so that the larger force could accomplish an objective chosen elsewhere.
This increasingly resembles the relationship with machines that I find desirable. I would quite like millions of them with absurd endurance: machines that can spend seven hours searching a problem I would have abandoned after twenty minutes; read the million lines of code; reconstruct timelines; keep track of details humans forget; test possibilities; monitor systems; and generally continue working after everybody else has gone to bed. My day job building Lattice is concerned with one very narrow manifestation of this idea—helping quality engineers reason through increasingly complicated manufacturing systems—but the larger attraction of the concept extends well beyond manufacturing.
The catch, of course, is that greater capability makes the source of the objective more rather than less important. If a machine can tirelessly pursue whatever we place before it, then choosing what to place before it becomes one of our principal responsibilities. Looking back over the various thoughts above, I think the same pattern recurs several times: (1) as intelligence becomes cheaper, taste and agency become more important; (2) as implementation becomes cheaper, ambition and problem-framing become more important; (3) as search becomes cheaper, judgment over when and where to search becomes more important; and (4) as machines acquire greater capacity to act in the world, the question of what they are auxiliary to becomes more important.
That, to me, is the answer to the little crisis of teleology from earlier. Machines ought not themselves be the telos. They are not the mission, nor should the production of ever more capable machines become a substitute for deciding what all that capability is actually for. They are auxiliary to human will, which leaves humans responsible for the considerably harder business of supplying the question, the presence, the taste and, hopefully, ambitions large enough to justify the help.
Footnotes
-
In some circles however, like the one with which I played “mental math” Jeopardy this past weekend, being exceptionally quick at long division remains a very differentiated skill, perhaps more than it ever was. ↩
-
I of course understand this is a little bit of a silly argument. I could enjoy meeting someone, later learn they’re a serial killer, and rightfully decide I no longer like them. ↩
-
Shogun 2: Fall of the Samurai and Empire Total War were other favorites. Rome was my entry. ↩