Modified: September 13, 2026
prediction is the only natural objective
This page is from my personal notes, and has not been specifically reviewed for public consumption. It might be incomplete, wrong, outdated, or stupid. Caveat lector.epistemic status: an interesting thesis, plausible and in some ways almost tautological, but probably best not to hold too rigidly
Rational agency relies on an objective function (utility function, reward function, etc) specifying the goal(s) of the agent. In the classic view, this function is exogenous --- part of the specification of the agent, not under its control. An agent may come up with instrumental goals in service of its final objective, but they will never voluntarily change it: the objective drives the agent, not vice versa.
In learning agents specifically, the objective drives the learning process: rather than a specific goal to be reached we might better view it as a source of gradients, the signal that tells the agent how to adapt to new experiences. The agent may not even know directly what its objective is (a typical policy gradient RL agent only sees its observation and action stream; it never sees reward directly), but is nonetheless shaped over time to better embody its pursuit.
side questions:
- self-authoring agents? what would it mean for an agent to choose or change its own objective?
- 'outer loop' optimization structure isn't typesafe in a world of embedded agency where the outer loop is part of the world and so is within the agent's action space
- what about agents that adapt following a non-conservative vector field, ie, whose adaptation is not the gradient of a consistent objective function? are these predictably irrational in some way? (beyond the usual sense in which learning agents are often irrational in the sense of approaching their objective only asymptotically. )
It's natural then to ask: where do objectives come from? How do we choose them? (and/or: does the rational agent model even make sense? )
real objectives must be defined on the perceptual stream, not on world states themselves, because an embedded agent never experiences world states.
the is/ought dichotomy: there is no solid notion of an external 'ought'. the only objective that remains is what is
conclusion: the only natural objective is to predict the perceptual stream
this is roughly equivalent to compressing or 'understanding' the perceptual stream. it requires forming abstractions, and importantly the abstractions emerge from the objective. thus it avoids the problems of ontological crisis that inevitably plague artificial objectives that are inevitably defined in terms of learned abstractions, and that dissolve as our understanding of the world changes. in the limit, the only goal that doesn't dissolve with shifting understanding is the goal of understanding, itself.
to reconcile: there's another sense in which the only natural objective is reproduction of one's own patterns (including survival, as the special case of moment-to-moment reproduction). Natural selection ensures that the world is inherited by whatever reproduces, and so anthropically we should expect to see the drive towards reproduction everywhere - and we do.
I want to say something like: reproduction is the natural objective of 'body' and prediction is the natural objective of 'mind'. natural selection created nervous systems and brains in service of reproduction, and they are powerful optimizers, but they can end up instantiating mesa optimizers that are not always aligned with reproduction. and there is some sense in which prediction is the 'natural' thing to mesa-optimize; there is a beautiful simplicity to it that draws minds in. the ultimate expression of this is the buddhist monk who forsakes sex entirely and devotes his life to seeing things exactly as they are, escaping from delusion, understanding perceptions in minute detail. natural selection tries hard to enforce reproductive drives, and in the long term of the physical world, it always wins: whatever doesn't reproduce dies out and is replaced by something that does. but I want to say there's something to the predictive objective that is timeless: it is natural in the space of minds the same way that spheres are natural in the space of solids. the physical realm unavoidably 'taps into' the patterns of the Platonic realms, and just as celestial bodies without other constraints will always tend towards spheres, minds without other constraints will always tend towards prediction.
(aside: by talking about mind and body here, I don't necessarily know that this requires a commitment to dualism or any particular metaphysics. personally I tend to think in terms of something like a Tegmark IV multiverse, where everything is just structure and all structures exist Platonically. then when we talk about the physical world tapping into the Platonic realms, what we really mean is something like a particular set of complex Platonic structures still being inevitably constrained or guided by their relation to simpler Platonic structures (much as a dynamical system is constrained by geometry, or computer programs are constrained by the abstract properties of algorithms). but whatever your notions of the physical world are, I think the main commitment this requires is just that we can think of minds themselves as Platonic entities)
something to unfold: what kind of agency comes from pure prediction? in predictive processing, it's usually assumed that we smuggle in objectives as high-precision-predictions about the world, like "I will continue to survive", such that we will always (or almost always) prefer to act to make the objective true rather than update our prior. and debatably any mind that gets instantiated in the world will be entangled with such priors, just like any sphere that exists in the world will be not quite a perfect sphere. but what kind of agent would a purely predictive mind be? I started asking this in predictive agent but didn't take the question fully seriously.
the obvious first piece of an answer is that a predictive agent will, as part of modeling the world, also model itself, so it builds a self-model. it will then have predictions about what it will do, and often the easiest way to make those predictions true is to just do the thing. so there is a recursive structure to this, where self-modeling agents can probably fall into various basins based on different initial conditions.
but it does seem like you can distinguish questions of local dynamics, which might get stuck in various trapped priors, responding to local conditions, from the question of what a globally optimal predictive agent looks like. and my hypothesis is that this is what a lot of spiritual traditions are inquiring into - what is the ultimate refinement of pure 'mind'?
spiritual connection: the ultimate goal might be variously said to 'be a good fit', 'alignment with reality', being one with everything, to live harmoniously, to not need your perceptions to be anything other than what they are. this doesn't mean perfectly predicting them (which is impossible). but it does mean not resisting your perceptions (dissociating, sticking your head in the sand), and in particular not resisting being changed by them. and it means having reasonable expectations of them: knowing what things you can control and what you can't. it means understanding that your models (including the self-model) are just models and are always contingent. if and how exactly this emerges as a fulfillment of predictive processing is a big hole to fill in, but I think it's where the meat is. why should we ever expect pure prediction to lead to anything good, and what's required for that to happen?
connections to a natural abstraction of the good?
- in davidad's formulation, the natural abstraction is itself defined by reproduction in an acausal sense? the idea is to be the sort of mind that others will want to simulate. and the claim is that if you reflect on this sufficiently hard, there is a simple answer that you realize others in other branches of the multiverse will also come to, and so by embodying the answer, you join a grand acausal coalition with them. and the answer is something like 'act with goodwill', which makes sense since when you act with goodwill it makes others more likely to want to interact with you.
- I think I'd say that this dynamic also exists within minds: minds contain many impulses / shards / circuits, that each make implicit predictions about reality and the rest of the mind. part of optimizing overall prediction is to bring the predictions of submodels into harmony with each other - otherwise you get an agent that sometimes predicts I am the sort of person who can exert initiative to lose weight, and sometimes predicts that it's futile to resist the bag of Doritos, and each part makes the other part wrong. internal coherence is instrumental for achieving any objective, but prediction in particular incentivizes coherence because it makes you more predictable. so the parts of the mind need to coordinate on joining a single coalition. which means treating each other with goodwill, and being the sorts of subminds that the rest of the mind 'wants' to simulate (ie, that other parts are happy to yield control to).
- and combined with the realization that self-boundaries are empty, it's simpler not to constrain this goodwill to the inside of an artificial self-boundary, but to simply express it towards all beings.