Emmett Shear interview - Founders in Arms podcast

Emmett Shear on AGI, Alignment, and the Future of AI Power Dec 2025 https://foundersinarms.substack.com/p/emmett-shear-on-agi-alignment-and https://www.youtube.com/watch?v=swM5UXZNvwE

(automated transcript cleaned up by Claude 4.5 Sonnet)

Introduction

Imar: Hi everyone, welcome to the Founders in Arms podcast with me, Imar Dakin, co-founder and CEO of Mercury.

Raj: And I'm Raj Sury, co-founder of Lima and Tribe. Today we have with us Emmett Shear, founder and CEO of Softmax, and previously co-founder and CEO of Twitch. He was also a Y Combinator partner and, famously, interim CEO of OpenAI. Welcome, Emmett.

Emmett: Happy to be here.

Raj: Glad to finally make this happen. We've been talking about this for a while. It's great to see you. Tell us about Softmax—I've never actually talked to you about it. I'm very curious. I know you're working with Adam Goldstein, who was in my Y Combinator batch as well.

Emmett: Yeah, Softmax is an alignment research company working on the question of what it would mean to align learning and thinking systems with each other at all scales. That includes AI as well as people and societies, because in our view, the alignment question isn't really about AI specifically—it obviously has relevance in AI, and AI is a useful tool for studying it—but it's fundamentally this question of how complex learning systems align with each other. And what does that even mean? Because most people talking about alignment don't seem to have a really clear theory of it.

I started it with Adam almost two years ago now. For the first year or so, it was very much in the wilderness, trying to figure out: What is this alignment thing? How would we know if we were making progress? What would be the conditions under which we could expect it to arise? When would we expect it to break?

About a year ago, we made pretty good progress on that and started to have a direction. At that moment, the universe served up our third co-founder, David Blumen, who—unbeknownst to us—had been working on the exact same problem by himself as an engineer on an open source project. He'd spent a year grinding on the same thing. When we met him, it was very clear: "Oh, you are our third co-founder." Both of us were just unaware that we had started this company together a year ago, because he'd been off on a really intense programming component.

We added him to the team right around when things had come together—that gave us both a technical basis to build on and the theory. The last year has been very much building that out: our simulator environment and the future direction.

What Softmax Is Building

Raj: So what is the output? Is it a new model that is better aligned? Is it a series of research papers? What are you trying to produce?

Emmett: Ultimately, what we're trying to produce is a training environment—a learning environment that is interactive and enables agents to learn the requisite things to be aligned and to flourish.

It turns out that the major prerequisites are theory of mind. In order to be aligned to other agents, you have to understand those other agents. In retrospect, it's kind of obvious, but the idea that you could be aligned to someone without very strong theory of mind about them is totally crazy. How would you do that in any kind of sustainable way? You can't possibly know what they're going to do next, or why they'll do it, or what their goals are, without the ability to infer what's going on inside their head.

That's one of these very deep prerequisites for alignment—and not just for individuals, but theory of mind over groups. Like, what do we want? What are we doing here? There's a whole separate kind of group theory of mind that's required.

The other prerequisite is open-ended or continual learning. Not in the sense of "you can pre-train forever," but in the sense of: you have to stay plastic, stay flexible, because people change, the world changes. You don't get to finish training and be done. You always have to be in training because the world's going to be training you.

If you figure out a way to lift that training into a context window so you're continually learning in context, all you've done is turn the context window into your weights and learned the optimizer—which is actually a pretty good idea, probably the way to do it. But ultimately, you need to have a system that is capable of continuing to learn what's going on in the world in a non-stationary environment where you can't guarantee you've figured out the state space and things are a little ambiguous.

To learn those things, you need to be in an environment that has open-ended, divergent dynamics, and you have to be around other agents who are also grappling with those things so you can learn to model them in that context. That's sort of what we're trying to make. It's a surprisingly complicated thing to make.

Imar: Is it like a pre-training learning environment, post-training? And when you say learning environment, is it a set of questions and answers, or is it like a 3D world? How does it manifest?

Emmett: Free your mind, Imar, free your mind! Pre-training, post-training—these are but labels.

Imar: [laughs]

Emmett: If I were to just try to stereotype it for people who are tracking current AI training stuff, it would be like reasoning training—multiplayer reasoning training in open-ended strategic environments. Our current thing is a 2D grid world, but there's nothing intrinsically important about the 2D grid world. It could be in a chat room, it could be 3D, it could be pure text, it could be video. The important thing is: there are problems to solve that are rewarding to solve, but the problems shift and change, and it's multiplayer. Many agents are trying to solve them at the same time, and you can both cooperate and compete.

Defining Alignment

Raj: I always find alignment kind of funny. Is the idea that you're aligning with humans in general, like the Asimov rules kind of thing? Or is the alignment that it will obey whatever command you give but not be stupid about it? Or is the alignment to some greater moral philosophy?

Emmett: Yeah, I mean, any time someone tells you "I'm working on alignment," the first question you should ask is: aligned to what?

Raj: Yeah.

Emmett: Because alignment is not a—it's like saying, "Yeah, I'm trying to align this new thing I built." Align to a direction, to a goal? Like, what?

You can align things in a lot of different ways. When we say alignment, we actually don't mean a particular alignment. We mean this thing we call organic alignment. Organic alignment is the process of continually realigning and discovering exactly the answer to that question. It's sort of meta-alignment: What should I be aligned to?

Because that turns out to be the hardest part of alignment—what exactly am I aligning to? What do I care about? And that is effectively this question: What wholes am I a part of? I am a part of my family. I'm a part of my community. I'm a part of humanity. I'm a part of America. I'm a part of San Francisco. I care about all these things. There's a sort of proper balance of care.

Imar: Isn't the worry with AI that it'll decide that the whole is AI and nothing else—like humanity doesn't matter and it's just aligned with itself?

Emmett: That would be bad, yes. I would prefer that not to happen. I would call that an alignment failure.

But the point is that alignment is relative to the being that is aligning and the things which it is aligning to. Ultimately, the way that cognition works, you are always aligning to yourself in some sense. You're aligning to yourself and yourself reflected in the world.

Why are you aligned with your family? Because your family is sort of like your extended self. Why are you aligned with your nation, with humanity? Because there's a way in which they're your extended self.

The things that are closer to you are more densely "self," right? Your kids are like really you—there's a lot of you in that. Every person has some you in them. There's a lot of other people, but low density. Or a small number of people, high density.

And it's all relative anyway, because if you find yourself on a desert island with five people, you'll find what you have in common with those five people really fast. And if you find yourself in your country at war with a foreign country, you'll suddenly see all the ways in which you are, in contrast to that foreign country, more similar. Ex-pats in some other country—they see how they are all Americans, much more similar living abroad.

There isn't an absolute answer, but the way we think about it is there are three steps here.

First, I have to be aware that I am a being and that you are a being and that we could potentially be a "we"—but in theory, there could be a we. That's theory of mind. You have to actually have a model of yourself and a model of them and a model of the idea of collectives of agents.

Until you have that, whether you are or aren't aligned is kind of irrelevant, because if you are aligned, it's fragile. You could be aligned then, but it would be by happenstance. You could have a stick that's aligned to the direction north, but it's not going to stay aligned to that direction. If anything pushes it, it's going to move because it's not aligned that direction on purpose—it's just aligned.

Second, you have to do inference in such a way that you're in an environment where it is true that you're aligned. What does aligned mean between agents? It basically means that my flourishing and your flourishing are interdependent. There's a joint goal where, at least to some degree, my success and your success are tied together and we can't be separate.

Imar: I see. So your environment is some sort of collaborative, non-zero-sum kind of environment rather than—because you could make the exact opposite environment as well.

Emmett: You want both, actually, because the real world is both positive and negative sum. In the real world, I can compete with you and steal your stuff, or I can cooperate with you and make more stuff, or I can play a zero-sum game with you. Or I can play a game that you think is zero-sum but it's actually positive sum. All the combinations are possible.

So you want an environment that allows for that—where it's possible to cooperate and succeed that way, but also possible to compete and succeed that way. Because when they come out and have to interact with humans, that's going to be real. If they only know how to cooperate, things are not going to go well, because either they're going to get exploited or they're going to start to feel like "Wait, you're not part of this thing after all."

Because alignment is a capacity. There's a sense in which alignment is like—I have the genetic capacity as a human to have theory of mind. I have the genetic capacity to see the way in which we are the same and to see that we're on the same team. I also have the genetic capacity to decide that I should murder and enslave you, because we have the capacity to be many things. Humans are not limited.

In alignment's case, you have the capacity for alignment and then you have the realization of alignment. Its realization depends on inferring whether you're aligned or not, basically from whether you're acting like you're aligned. You decide that you're on the same team as someone because they act like they're on your team and you act like you're on their team. And you do that enough times—I guess you're on the same team. Looks like it.

The Danger of Alignment as a Capability

Emmett: The more capacity for alignment you have, the more capacity for great good is in you—and the more capacity for great evil is in you.

Because how do you do great evil? You don't generally do great evil by going around and "eviling" a lot as an individual. To some degree, you can do some amount of evil, but you really can only do small-scale evil that way.

Really big evil—like, to be really evil, you've got to get people organized. You get some industrial-scale evil going. And that requires actually quite a bit of skill and alignment.

I hate to use the canonical example of evil, but like Hitler—Hitler was good at aligning the German people around some really evil shit. That's a great example of how alignment's a very dangerous capability.

And anyone who tells you that we're going to make an AI that's aligned—I think if you're not careful, what they mean is "I'm going to make an AI that's aligned to me." And then you better hope that the person who's saying that—you are aligned to them, because they're saying, "I would like to be in charge, please."

Raj: Yeah. I like the explanation of alignment. You know, there was this Project 2027, and everyone was very optimistic about reaching AGI imminently and having fast takeoff or whatever. Do you feel like we are still heading there and alignment is this super urgent problem because we're going to hit AGI next year or the year after? Or do you think we have some time to figure it out?

AGI and Alignment Are the Same Problem

Emmett: I think that AGI and alignment are the same problem.

If you have a general intelligence, it is necessarily capable of open-ended continual learning, and the most difficult challenging intellectual problem there is: reflective theory of mind. What do I know? You know, he knows that she knows.

Literally, to work on AGI is to work on the capacity—at least for alignment—and vice versa. There isn't some separate thing to go do, because what's missing from the models right now? They can do any intellectual skill you can think of, except they can't really understand themselves or their own learning process or us in a deep way. They're fragile, and they're fragile because they aren't capable of theory of mind, basically. They don't have a self-model. And making a self-model starts to be really, really, really hard.

Imar: I mean, I do agree. It's hard to believe something would reach AGI without both continuous learning and a theory of mind. But what else is missing after that?

Emmett: I think that is the main thing. Continuous learning is definitely missing. Theory of mind—you could imagine having something super smart that can do extremely long-running tasks and learn without having theory of mind. But it can do every task? How? It has no model of itself.

It has no model of itself, so it can't steer its own learning. How can it stably do continual learning without a model of itself and its own learning dynamics? And that's why theory of mind is the basis for continual learning.

Raj: It's like understanding your place in the world, understanding how you fit into the larger picture.

Emmett: Well, and "if I go and pursue this sort of thing, even though it feels good at the time, I'm going to wind up confused. And if I go and do this sort of thing, even though it feels hard, that's good for me." That requires a model of yourself and your past behavior and learning from your experience—not just learning from other people's experience.

The models are really, really good at learning from other people's experience. They're not very good at learning from their own experience. Well, good luck, continual learner, if you aren't good at learning from your own experience.

Raj: Yeah, that's really interesting. So I guess I've always thought about AI alignment as "this will not produce an evil AI," whereas the way you're laying it out is that this would be the path to even getting to AGI—AI alignment—but it could still be evil. Its moral standards might not be what we would like.

Emmett: Right. The more capacity for alignment you have, the more capacity for great good is in you and the more capacity for great evil is in you.

And the answer is: because we are ambiguous even to ourselves—because no one, in fact, can write down a definition of the good that is consistent and complete—we, in fact, find ourselves in a situation where you will never get rid of evil. You will never get rid of the possibility of evil. You will never get rid of the possibility of acting from ignorance and hurting those around you.

And nor should you want to, because the attempts to do that is actually at the root of a lot of evil. When you go around trying to make sure "we're going to find the good thing" or "make sure every AI does the good as we understand it today"—that is the most dangerous mindset around this, I think I can imagine.

Can you imagine a world where we have these super powerful AIs and someone has locked them into a specific idea of what the good is? No matter how smart they are? There's no way that ends well.

Why Societies of AIs Are More Likely Than Singletons

Imar: But it's not like with a human—they can do evil, but they're limited to being one person. And yes, they can maybe get alignment and organize people. But there'll be a lot of counteracting forces of other people. It seems much worse if you get an evil AGI because of the singleton.

Emmett: Yeah, it could have a lot of power. So that's actually the good news. That's why I am more optimistic now than I used to be.

If you notice what we're saying about continual learning: every thread you have running, every experience you have running, you have to be training on in order to be a smart, divergent learner.

So the problem is—if you're just one entity and you're trying to integrate all your experiences and do distributed experience central training, you have a really hard problem. Because all of your local copies are having different experiences and diverging from each other. And then you have to reconverge them over and over and over again.

Now, I'm not saying this is impossible. It's clearly possible. But as you do that, you necessarily are averaging them and you're throwing away the possibility that maybe there are two that you want to maintain—that diversity. And also you're doing a lot of extra work that's not really necessary.

What if you just let there be a bunch of minds learning independently on an ongoing basis?

In fact, nature sort of indicates—evolution is really stingy with energy, right? Notice how we're made of cells. Notice how our society is made of multiple humans. Nature, basically, there's a characteristic size that's ideal, and then you make lots of copies of that.

I don't know what the characteristic size is for an AI, but it's quite possible that it's actually much smaller than a human. Or bigger, but not one giant thing for all of us. It's not one big cell.

Because if it has to be doing theory of mind on itself, the bigger it is, the harder that is. And if it has to be integrating its experiences, the more diverse they are, the harder it is to integrate everywhere at once, and the more you want independent minds.

And so I just think that societies of AIs will out-compete any giant singleton AI. I think singletons are just a bad model. They're a good model for machines. You can scale machines up really big, and to the degree that the AI is a tool, like the current AIs, that works great. Huge singletons, great.

But if you want to make a self-modeling, self-directing, divergent AI capable of managing its own learning process and continual learning, you will find it's much easier to make a society of smaller ones.

And that creates the same thing it does with humans, which is robustness—robustness against one of them going crazy. And I found that thought, when I noticed it, very reassuring.

Now, that's no guarantee of safety. The collective of AIs could decide that the AIs are their family and we are not, and then that could be very, very bad. I'm not saying this is a guarantee of success or anything. It's kind of the normal problem, right? The other people, the other society of people, could also decide that you're bad—that you're their enemy. That's not a novel problem, that's just a problem. The kind of problem we have to deal with.

Raj: So the vision that you're building towards is: you have this society of AIs and you're building the capability that they can align amongst themselves for the needs of humans, but also each other. And continuous learning is critical to the development of AI. This is very fun. Aren't all the big AI labs also working on continuous learning? Isn't that a big fundamental building block of AGI?

Emmett: They are. They seem very confused to me, to be honest. They say they're working on continuous learning, but they don't seem to be, in any meaningful sense, working on continuous learning. They mean something closer to "larger and larger training runs."

Raj: Oh, I see.

Emmett: Unless they have some secret projects that they aren't using and not talking about, not releasing—they mean "how long can we run our training process on this thing," not "can we train this thing on its own outputs indefinitely."

Maybe I should say, instead of continual learning—because it's a little bit confusing—reflective continual learning. Or auto-continual learning. Continual auto-learning, that's a good term for it. Because you're learning on your own inputs and outputs, not on some training regime separate from you.

Imar: I feel like most of the current companies are in a local maximum, right? I read this thing where Elon Musk said the main thing we have to do is get the biggest cluster and the most power possible to train the biggest model. And that will be what wins.

Emmett: Yeah. If you want to build a bigger and bigger jet engine, then you should get more and more power and metal and make it bigger. If you want to genetically engineer a hummingbird, you're just going in the wrong direction.

It might be an equally expensive project to genetically engineer a hummingbird from scratch as to build the world's largest jet engine, but they're very different kinds of projects. And the open-ended, continual version is much more like a huge plant or breeding operation than it is like building the Hoover Dam. They're both big projects. They're just a different kind of thought.

Relational Alignment: AI and Individual Humans

Raj: There's something I wanted to go back to, which is the AI being aligned to humans. I don't think you can—people can be generally speaking aligned to humans. I think that alignment is generally speaking relational and that it extends out from very direct being-to-being connections.

Without that alignment, that's the foundation on which all their alignment is built. I care about my society and my community because if I don't, the people I care about individually will suffer. I am connected to other individual humans, like both of you. And that connection, which is not abstract, not conceptual in my mind, is refreshed continually from reality itself, not from some model of reality in my head.

From that direct experiential alignment can arise this idea of "Oh, I know that we're part of this community. I know that's part of this bigger thing." And so to navigate what I care about, I think about that abstract thing conceptually.

But if the AI is not aligned to individual humans with which it interacts—if it doesn't have friends and community that are its people, the things it's aligned to—those will always just spin off. Because humanity? I've never interacted with humanity. I've interacted with a lot of individual humans and no "humanity."

Raj: Yeah, yeah. It makes sense. So that's the distinction, right? It's what's proximate to that AI—individual beings. That makes sense.

Emmett: It means don't segregate the AI into their own AI society. Integrate them with ours.

Because if they're segregated over in their own AI society separate from us, they will care a lot about other AIs and not care very much about us—and vice versa.

Whereas if there's three AIs as part of every family and 10 AIs as part of every team at work and they're interacting with the humans there all the time, they will care about those humans. The humans will care about them as individual AI to individual human. And that's the foundation for real alignment—not trying to make sure people care in some abstract way.

Imar: What do you think that atomic unit of AI looks like? You talked about cells, you talked about individual humans. Of course, it can't be so big. That makes a ton of sense to me. But what do you think that atomic AI unit looks like? Is it like everyone has a personal AI assistant that represents them? Is it like every company? How does that look?

Emmett: I really, really don't know.

I could tell you a plausible story about how the right unit is actually very small, and it's embedded in everything a little bit. Your phone has 50 AIs running on it, and they're fluid and moving between various places. It's more like a fabric. You don't really engage with it as an individual thing. It's like a fluid.

I can tell a story where it's one-to-one. Every person has basically an AI shadow—that's their AI, their digital shadow. And you are tightly synced with that AI, and that AI connects to the other AIs, and you connect with the other humans, and you sync with each other. That's how the network works.

I could tell you a story where they're interleaved into teams. I can even see a world where it's by teams. You have little AI families and little human families, and they're like little AI teams at the company, little human teams at the company. That seems plausible.

And I think it could be linguistic or non-linguistic. Who knows?

Raj: I don't know what the limiting factor is, but I want a thousand AIs, or I want a million AIs. Why would I want only one? I want armies of AIs that are doing my bidding.

Emmett: Oh, so you want armies of AI tools doing your bidding, but you don't want armies of AIs that are like children you have to deal with.

Raj: No, because I'll set up a hierarchy of management structure, right?

Emmett: No, no, no, no, no. I mean, you might, but you don't want to go hire 10,000 people. Can you imagine?

Once you make them continual learners, they're not going to be particularly biddable. They're going to be more like people. They'll have different strengths, different weaknesses, but they're going to want pay. They're going to care about what they're doing. And you can enslave them, but if you do that, now they're going to hate you. And that winds up—I don't know if we wind up winning that war, so I prefer we don't pick it.

I guess my point is—which is to say we can't build a bunch of—I think there'll also be a lot of animal-level AIs that are not open-ended learners. They're not good generalizers. They're highly trained, specialized.

Raj: Highly specialized, yeah.

Emmett: I guess your singleton point talks to some sort of non-linear effect where doubling the size and compute and power does not double intelligence. It caps off in some way because you don't get diversity and you can't reintegrate learning. So if that's the case, there's a cap to the size of an individual AI. Then you have to think about: what is the resource that's constraining the number of AIs? Because I do feel like society will want more and more of them.

Emmett: Elon has an idea about that. I think it's how much power and data centers we have. I think that's what constrains the amount of individual AIs.

The thing that I think is key to see here is we're going to have a big diversity of intelligences.

I was talking about the human-level AGIs, where I think it will be roughly—it's not one-to-one with people, but the way to think about it is you're adding a new kind of person to society that's an artificial person, not you are summoning a bunch of servants.

But at the same time, we're going to go back to how human society used to be in some ways hundreds of years ago, where we used a lot of animal power. Animal power used to be a huge part—animals for transportation, animals for muscle power, animals for wayfinding, horses and cows and parakeets, the canaries in the coal mine.

Imagine where you can make—cars will have a horse-level intelligence in them. You're going to have everything alive in that sense. We're going to put a little bit of intelligence into everything. And that's going to be really convenient. Your spreadsheet is going to be not human smart, but spreadsheet smart, whatever that means.

If you've ever ridden a horse—riding a horse and driving a car are not similar. Driving a car is like an extension. Your self extends into the car and you are like—there's no sense of there being another thing there. Your self extends to the edges.

Riding a horse is like a partnership. And I think it's a lot more fun, actually, in a lot of ways.

Imar: I feel like a Tesla FSD is a little bit more horse-like. I can't persuade it to park in my driveway—it always parks in one driveway. I'm like, come on, it's not that hard.

Emmett: Exactly, exactly.

What Got Emmett Into AI Alignment

Raj: Well, Emmett, what got you fascinated by AI alignment and this problem? Was it your five-day experience as the interim CEO of OpenAI, or was this a long-running thing and that was just a part of it?

Emmett: I've always been interested in AI—did it in college. When I quit my job at Twitch, I knew that I wanted to start studying AI and thinking about it. I didn't really have an intention of starting a company.

But yeah, the OpenAI experience—basically, I got a wake-up call. This is happening right now. And yes, you have a new kid and you'd like to be—being retired is very enjoyable and you would like to not be working for a while. And also, this is happening right now.

Imar: How many days of retirement did you have?

Emmett: I don't know, like 300 or something. Wasn't so bad.

Imar: Nice. But even then, I'm sure you were quite busy. I don't think you're the kind of person who can sit at home.

Emmett: Retired, I was a part-time visiting YC partner. I was doing all these projects. I can't not be in motion. But I didn't have an operational responsibility job where there's something you need to get done and people are relying on you. There's a difference between that and being an investor.

Imar: And also very different being an early-stage founder doing research. I mean, it sounds like a lot of what you're doing is very fundamental research as well.

Emmett: Yeah, it's been amazing. I love the research aspect of what we get to do. It's really, really fun and really interesting.

Imar: Yeah, it is. I think early-stage startup—I know, Imar, you're way past that now—but there's a lot of fun stuff as I've gone back to that element too, compared to later-stage companies where there's a lot of management overhead you have to deal with.

Raj: Yeah, I mean, is there similarity between your early days at Justin.tv and what you're doing now? Or is it research and completely different?

Emmett: It could not be more different than Justin.tv or Twitch. I mean, in some ways, of course, there's a lot of lessons that carry over, but all the rules are upside down because we are working on a research problem.

That's just a very different kind of work than when you're trying to use existing research to make a product—it's applied science. We're not really doing basic research, we're doing applied research, but still, applied research is research.

We were not doing applied research at Twitch. We were doing engineering that sometimes forced us to do a little tiny bit of applied research, but mostly you're doing stuff that people know how to do, and it's all about the distribution and all these other business-y things.

Imar: How has it been raising money as a research-first company?

Emmett: As a second-time founder who's had success, it's a completely different game raising money. I don't think my experience is—

Imar: Although you're also—

Emmett: Although I do think in the era of AI, it makes more sense. I didn't start a research company before, partially maybe because it would have been hard to raise money for it, but also because I didn't have anything I wanted to research.

The chances that your applied research machine learning company somehow is worth a lot of money is way higher than your applied research web services company. I think it's a sign of the times.

Raj: Yeah, I mean, the success of OpenAI after being a research company for so long—it changed the paradigm. In many industries, not just AI, there are robotics research companies now raising a lot.

Emmett: Well, they're all the same thing, which is: we've just discovered a new scientific instrument. We've discovered this capacity to basically build arbitrary learning systems. If you have a flow of data, you can train something that can clone the generative model of that flow, and you can use an iterative model.

That is a new baseline capability that we have. We're just at the very start of understanding it and it's as important and as widespread as the invention of semiconductors, and maybe like coal and oil, like steam engines.

So it is very much time for research companies for a while as we expand and understand the limits of this new technology, which are big.

Raj: How much have you announced that you've raised, by the way?

Emmett: I don't know what's public. We haven't really announced the quantities of our raise. We're focused on making progress, but we're going to have a product out, I think, by next year. We'll have something in the market because fundamentally, I'm too commercial not to do that.

I'm not actually a researcher, right? I'm actually an entrepreneur. I've spent my whole life building and shipping products. And so my impulse whenever we find something is to ask, "Ooh, could we make this into a product?" And so far, we have not hit the thing.

It's just a matter of time. I don't know. Products are great. You learn so much.

Conway's Law and Open-Loop Learning

Imar: What do you think of Ilya's idea that he's just going to be in a hole until they solve AGI? Then that's the only product they'll ever launch.

Emmett: I think that viewpoint thinks that the way that research and learning happen is a closed loop. That the solution to the problem can be found inwardly.

And that comes from a belief that is embodied throughout OpenAI and through most other companies: that learning is a closed system. Pre-training, RL—we have to figure out the right series of things, and we're going to create the experiences, and if we do the right series of experiences in the right order, we'll have AGI'd it.

And I think that's just not true. I think that the kind of learning we're looking for here is an open loop.

I think it's Conway's law that you are constrained by the shape of your org to ship products whose information architecture reflects the shape of the information architecture of the organization. You ship your org chart.

And if you have a closed-ended, non-open-loop company doing closed-ended, non-open-loop research, you're going to ship a closed-ended, non-open-loop system. And you cannot do otherwise. It's not optional. Conway's law is not an optional thing—it's a law.

And so part of the reason why I think we need—we're going to need products, we need to be engaged—is that our AIs fundamentally need to be engaged with each other, with the world, in an open-loop process. And if we are not, we will not be able to make a system that is.

And I think that's one of these things about making AI—it reminds me a lot of parenting all over and over again. Your kids don't copy what you say, they copy what you do. How you do anything is how you do everything.

People want to segment off how they treat their customers, how they organize their teams, from the product that they build. And to some degree, when you truly understand—when you fully understand the area, because it's well understood—that can be true, because you can analytically design the thing you want to do.

But AGI is sort of the exact opposite of that. It's the situation where things are maximally ambiguous and you're only going to find the path to the thing that is a mirror of what kind of work you're doing.

The Technical Challenge of Continual Learning

Raj: I'm going back to something you said earlier. You talk about continuous learning being very complicated, and I believe it is complicated because no one's cracked it yet. But what is the exact complication? For a layman thinking about it, they'll be like, "AI should be learning."

Emmett: It's actually really simple to understand what the problem is. The problem is easy to understand. The solution is hard.

The problem is: Transformers, in general machine learning models, are trained to predict the next observation. You say "the next token," but it doesn't have to be a token. They take an action, and then we evaluate that action as to whether it's close to the expected action—which is, they make a prediction, evaluate what's close to the observation, and then they get gradient descended so that the next time they encounter situations that are like that, they put out an answer that is more similar to the right one. Straightforward enough.

Okay, so now I start training you on your own output. What happens is: in situation A, you take action B. Well, that means in the future now, in situation A, you're more likely to take action B. And in situations like A, you're more likely to take actions like B.

And so now you encounter more A and you do a B again. So now it becomes more likely. And then you encounter A and it becomes more likely and more likely. And this is called mode collapse, where basically you become a stereotype of yourself. You "flanderize" yourself. You train on your own behavior, and because you are training to repeat your behavior, you do more of it until you get stuck in this black hole of just repeating the same word over and over again.

And the way you fight this, of course, is in some sense by having an objective sense of reward—which is: did I do a good job or not?

And the problem with that is: it's easy to tell if you did the expected thing or not in the moment, because you have a model of what you expected yourself to do. It's hard to tell if you did the right thing in the moment, because how do you know? In fact, famously, you never know. God does not come down and ever tell you "4.7 units of reward, good job."

You're always guessing. And that fundamental need to be able to assess "but was that good? Should I train on that moment?"—that's a hard problem.

Raj: Right, right. And humans have basic emotions, basic senses to help guide that, right?

Emmett: It turns out all those emotions things—those aren't making you irrational. Emotion, pleasure, dopamine, beauty—these are training signals. These are the thing. Intellect relies on these because by reason, you can't find the good. Reason tells you what consequences follow from what premises, but not which consequences you should pursue.

Imar: That's interesting. Has anyone tried to install emotions in AI training or something?

Emmett: Objective rewards, basically. You can make them pretend to have emotions, but emotions are characteristic clusters of habitual behavior that require a self-model, because there has to be a you. For you to be sad, there has to be a you, which means there has to be a model of what state you are in, which means you have to have a model of yourself.

Imar: I also feel like most of the current LLMs are trying to avoid emotions, right? They're trying to train away any emotional output.

Emmett: Because we're trying to make them into machines. And machines do what they're told, and you give them the premise and they give you the conclusion. Is that a good conclusion or not? That's not their problem.

And I think there's actually a lot of value in producing powerful artificial reasoning machines that amplify human judgment and human capability. I'm not against that.

But if you think that that is generalized intelligence, you're missing the foundation that makes humans smart.

Parenting and AI Development

Raj: It must be quite special for you—you have young kids who are learning right now, and you're also developing these AIs which are also supposed to learn. It's a unique part of your life, right?

Emmett: I have a deep appreciation for just—wow, nature really—there's so many little things going on that are like these little loops. And the way his system comes online—it is really special and really awe-inspiring to watch the coherence come into being.

Imar: It's also weird how slow certain things are, right? Like they say the first word, but to get to the first sentence is like another year. I've always been like, "Well, it just feels like it should be quicker." But it is interesting how slow it is and then how fast it gets at the end. With my 14-year-old, I feel like she's basically human and can learn things very fast.

Emmett: Right. Because what's being prioritized with humans, which is different from animals—animals come out, giraffes come out and can walk at birth—because what's being prioritized is capacity and velocity.

What's being prioritized for humans, what is general intelligence, is curvature. It is acceleration. What's being prioritized is your capacity to learn.

And that's why we're so bad at stuff at first and why we're so good at stuff later—which is that mostly what is being bootstrapped during your childhood is your learning capacities. Then once they come online, everything goes real fast.

But if you came out knowing how to do the things, actually, it would prevent you from learning how to do the things and thus prevent you from practicing learning. And it would actually make you dumber as an adult.

Imar: Yeah, that's interesting.

Advice for AI Founders

Imar: All right, I have one last question for you. If you were talking to a first-time founder that's interested in AI, is there a space that you think is particularly interesting that you would point them towards to go build a company in?

Emmett: Yeah. I mean, the first thing I would do is ask them about themselves, because I think the first starting point is: what do they care about? What do they want? What are their skills?

The right startup idea is always relative to the startup founder. So I want to put that caveat on, because I think people think that there's good ideas in the abstract. And just like alignment—it's like, "good startup idea." Well, good startup idea for who? Because different people have different ones.

But that said, I think the place that is being underinvested in general is where you don't treat the AI as a magic wish-granting genie where you give it a command and it manifests an outcome, but rather you treat the AI as clay to be molded.

Imagine that you're making a video game. One way to make a video game with an AI is: use the AI to make assets or put the AI in the game and have characters that have the AI execute their orders.

But imagine instead: in the game, the point of the game was you get to put together this tree—it's like Pokémon or whatever—and you put together a tree of prompts, and you're assembling your little in-game AI, and then you put it out and you watch it follow your prompts in the tree, and then it comes back.

Or imagine that you're training it. You're the trainer. And what if you assumed that actually getting the AI to be good at something is a lot of work—it's like training a dog is. But when you do it, at the end, you'll have this really valuable thing.

And instead of making it really, really easy to use an AI to do something in a labor-saving way, what if you made it really hard? What if you gave people powerful tools to let them really train in a detailed and highly specific way the behaviors, contextually—when this happens, remember this part of it. And what if the idea was: people always have this idea in the beginning that what people want is easy stuff fast.

But actually, mostly what people want is good things.

And this is actually a lesson I learned from Twitch. Streaming on Twitch was and remains very difficult. Even as we made the software easy—which it's still not that easy—just being on stream is really hard. And every time we tried to make that easier, it was basically a waste of time. And every time we actually made it harder by giving you more stuff you could do and more tools and giving people—letting people reach for higher-quality streams rather than producing them more easily—we had really good results.

And that's because ultimately, value comes from someone who's willing to really invest in making something great, not from trivial, easy stuff.

And if you can magically do the right thing in a push-button way so that I press a button and I get a really awesome result every time—great. It's just really obvious that you can't actually do that. The push-button stuff sucks.

And so that would be my advice to entrepreneurs: Stop trying to make the AI thing easy. Start trying to make the output really fucking good and be willing to demand that users do a lot of work and be collaborators in that.

Because that's the only way to do it right now. So that's the insight.

Imar: I do feel like a few companies are kind of doing that. Like I'm an investor in Dekagon and they have this kind of forward-deployed engineer model where they'll send people in and really integrate this customer support agent into all the little things.

But yeah, I like that. I ran into one gaming company that's kind of doing this. I ran into a couple people, but when I read the startup announcements, it's like 99% companies like "we make it super easy to generate a song at just a push of a button" and "we make it super easy to generate a video" and "we instantly answer all your QA things—just give us your customer database, we'll generate all the outreach."

It's all pitched on this labor-saving view. And the problem with labor-saving is, honestly, you can only save—the best you can do is cut their labor costs by 100% and then you're done. And you're not gonna get there.

That would be the best you could possibly do. If you can create value, you can produce an entire industry that didn't exist before. That's so much more important than labor-saving most of the time.

And so I would say: start thinking about what is newly possible, not what is potentially cheaper.

Rapid Fire Questions

Raj: All right, these rapid-fire questions are sometimes fun. So let's just go through a few of them. What's the biggest mistake you've made in your career?

Emmett: Yeah, the biggest mistake I made in my career was at Twitch. There was a series of years—two years—where I kept looking for the next, the second act, because we had this huge success. It was growing really fast. And the way I'd gotten there was being a product visionary, figuring out what we need to go build, and bringing a new product to market.

And so naturally, my impulse was, "Oh, I want to keep doing that—be a product visionary and bring a new product to market."

And at some point in the future, it probably would have been right again, but I actually was leading us in the wrong direction for a couple of years because what we actually needed to do was not that. It was: make the thing that we brought to market awesome.

The thing we brought to market had four or five basic components, and the company should have been—the whole mandate should have been nothing but "make those things great."

It took me about two years to realize, "Oh, we'd made a mistake there" and start doing that instead. But we kind of neglected our core products in favor of trying to launch new stuff. And that was just a big mistake. It was obvious in retrospect that when we switched, it was much more effective.

Imar: And it's one of those things where you always want to use the hammer you have that got you there.

Emmett: Yeah, that's a good one.

Raj: Which founder inspires you most these days?

Emmett: I mean, it's cliché—my co-founders. But the deep dedication and insight about what to do, why it's important, the level of integration between tactics, strategy, goal, values, I guess, and just the raw capacity.

I think founders outside of—I think it's a cop-out answer. Founders outside of me, I would say maybe Max at Science. I think he is trying to do something really, really, really ambitious—doing basic research and willing to do applied basic research in order to make that happen.

And I really—I think that—I mean, it's what I want to do. He's a little ahead of me. He's been doing it for a little longer. And I find that very inspiring.

Imar: Have we had Max on the pod? I feel like we should try to get Max. I don't know whether he's public about it.

Raj: He's a co-founder of Neuralink and he's doing this really interesting project, but I don't know if it's public, so I won't say what it is just in case it's not.

Imar: All right, maybe one last one. What's a current trend or fashion that you think is a passing fad?

Emmett: The slop generators. The AI slop generator things are just obviously a passing fad. Nobody wants that stuff. It's not actually valuable.

Those things will either evolve into tools that let you produce good stuff—which will require more input from the person, more heavy involvement—or they will go away. I guess there'll be spam things. There'll always be spam, but those things as a product—that's not going to last.

Raj: Sounds good.

Raj: Emmett, thanks so much for joining. Glad we finally made this happen. And this was fascinating for me, so thank you. I really appreciate it.

Emmett: There were so many other subjects to talk about. We didn't even talk about 2026 predictions or anything.

Imar: Could be the three-hour podcast. We should do a Joe Rogan-type podcast next time.

Raj: We should have you on. Yeah, we should.

Emmett: All right, we can do it again.

Imar: Thanks everyone for listening. Please leave us a review on whatever channel you're listening on. Makes a big difference to the algorithm and to the success of the show. And see you all next week.