long context is social

if we can only hit a fixed context window with transformer-like attention

and/or if some tasks require test-time training --- using extra compute to learn 'motor skills' or to jointly compress observations with existing concepts

(and the moment we have continual learning, every task births a new agent. every task creates a new context to learn from. )

then now we have differentiation between agents. we can't build a monolithic agent that sees everything, because context size is limited. maybe we could build a monolithic agent that compresses everything (that was done by subagents), but we won't, because the compute to jointly compress stuff also grows quadratically expensive (to compress N things, our network has to have size O(N), and we have to do O(1) backwards passes to remember each of the N things, so we're now at O(N^2) compute). unconstrained interaction is fundamentally quadratic, whether you implement it through attention or joint compression. so there is a limit on unconstrained interaction.

(just training the monolithic agent from the traces of a subagent also doesn't guarantee that the monolith learns the skills of the subagent. the subagent was trained on-policy, with feedback from the environment. any training that reuses this experience is going to be off-policy and can't be validated without further environmental interaction. so the specific skill artifact of the subagent is valuable)

thus, at some level of scale, we get a society of mind - intelligent behavior implemented by multiple agents interacting.

this is already showing up in Claude Code. We use 'subagents' to avoid blowing out the main context window.

it's also the case for people, in a different sense. we have to leave notes for ourselves when working on large projects because we can't keep the whole context and intent in memory at once. we are effectively "collaborating" with our past and future selves.

there is of course also the physical argument - a given volume of physical space can only store so much information, and communicating with other storage volumes elsewhere is necessarily slow (as in the cache/RAM/disk/network hierarchy)

(internally, we get agent separation from different contexts in trauma: some specific experiences are so painful that the 'main agent' does not want to integrate them, so it shuts its eyes and pretends they don't exist. when we have internal conflict, experiences we can't reconcile, maybe this is different coalitions of partly integrated contexts. the limit here on integrating contexts is not memory, compute, or physics, it's emotional: the parts don't want to have to deal with painful stuff, and are afraid to change and die)

differentiation between agents is what we get when we have different contexts and constrained bandwidth (so we can't transfer the shared context) and/or compute (so we can't jointly attend to the union of the shared contexts, even if we had it)

(we could push back on this easy conclusion: do the different contexts necessarily have different identities associated with them? if data are stored in a tree structure, is that inherently a 'social' process with each node in the tree a different being? or can it just be an algorithmic / hierarchical process, where the components implement fixed interfaces? this is ultimately subjective, a question of where we choose to take the intentional stance. you could see it as a) very permissive view: data structures (and OOP in general) are social, just with extremely rigid/brittle conventions. it's a 'society' more like cells than like people, since the parts aren't individually self-aware, but they do still have an informational boundary and do still have to 'cooperate'. I'd say this is an extreme view --- cool to contemplate but not the most parsimonious way of understanding things. b) less permissive: a society requires recursive self-modeling. the parts need to have models of the goals of other parts, and of their own goals. a tree of LLMs could start to have this, given an appropriate prompt explaining the situation. )

once we have multiple agents, we're now in a setting of negotiated social interaction. as Emmett Shear points out, social interaction is the essence of alignment. can we cooperate with people - and more than that, can we see ourselves as part of a "we", a shared entity?

self-alignment:

there are tensions even in cooperating with your future self --- do you spend time now writing extensive explanatory comments, designing maintainable code, paying down tech debt, etc., or do you take the quick win and leave the problems for your future self to figure out?

your future self is "you", but it doesn't necessarily share your goals: goals are (at least partially) derived from context, and it has a different context! in fact, you may have a hard time anticipating what goals they may care about. part of good software design is anticipating what sort of features you might want to add in the future; inevitably we get this wrong and have to update (or just poke holes in) our original abstractions. so even in the setting of 'solo' work, there's a question of alignment.

self-alignment isn't fully social, though, because you never interact with your future self. the communication goes entirely one-way. you can never talk with your future self, receive feedback from them, learn about them. of course, "you" do eventually learn about your future self, but then it's future-you doing the learning!

implications for research:

doing big tasks will require LLMs to learn social skills?

converse: to induce social skills, it's sufficient to incentivize LLMs to do big tasks?