I want to change the world. What does that mean? Suppose I create a billion-dollar company. That's an enormous amount of value. It's many…
tags: created: 2023-12-07 modified: 2023-12-07 References: https://arxiv.org/abs/2006.16236 The usual [ transformer ] [ attention…
A linear time-invariant system is one where the dependence of the output on the input is: linear: an input produces an output , and…
podcasts and audiobooks Alan Watts 'out of your mind' lectures
I've heard it said, and it's been ringing true to me, that the thing to do is live the life you want to have now , not plan to spend years…
Some things are genuinely hard to do. But many others I don't do just out of laziness, or maybe lack of [ agency ]. I know that they're…
Sources: https://www.lesswrong.com/posts/y5GftLezdozEHdXkL/an-intuitive-guide-to-garrabrant-induction https://www.lesswrong.com/posts/jtMXj…
A atomic orbital with two electrons both attached to the same atom. In contrast to a bond, where each atom contributes one electron. Lone…
Notes on https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context/ 'Standard' transformers have O(n**2) complexity…
There's a tendency to focus on things that we have the (conceptual/mathematical/societal) tools to understand, even when we know this is…
How do I justify being only 'mostly' [ vegetarian ]? I know that cows and chickens are abused to produce milk and eggs. Why is avoiding…
What does it mean to [ love ] someone? Of course this question has as many answers as there are people, and probably more. But here's one…
What does it mean to love someone? Of course this question has as many answers as there are people, and probably more. But here's one view…
see also [ metta ]
I saw this phrase on Twitter somewhere and it really resonates as a description of the ideal approach to science. There is no real…
A macrostate in statistical mechanics is a collection of base-level states; equivalently, a subset of [ phase space ]. It's what you see…
Mahamudra means the 'great seal' or 'great gesture'. We take and [ hold the view ] that each event arising in [ awareness ] --- every sight…
Managing for high-variance / creative work versus low-variance consistent work: https://blog.sbensu.com/posts/2023-01-18-high-variance…
There are a lot of difficult decisions to be made in life. Maybe you need to decide the business strategy of a company, knowing that good…
An idea I got from [ John Higgs ]'s discussion of metamodernism is that taking [ all models are wrong ] to its logical conclusion requires…
Sometimes I've been scared of losing my identity. In particular I worry about working a non-research job, or having sex with (or being…
I have a private theory about what marijuana does. I'll try to articulate it here. I don't know much about the public theories, so maybe…
A martingale is any [ stochastic process ] that stays the same in expectation. Formally, is a martingale if This condition is related to…
How to think about giving a good massage? Know which way the muscle fibers go. For deep release, exert force perpendicular to the muscle…
Reviewing this 3blue1brown video: https://www.youtube.com/watch?v=O85OWBJ2ayo The matrix exponential is written as E to the power of a…
The Woodbury-Morrison-Sherman matrix inversion lemma, is sometimes useful just for algebraic simplifications. In cases where and are…
Notation for Matrix Multiplication Let and . Then just by the definition of matrix multiplication (the summation over is performing…
References: Hu, Yang (2022) Feature Learning in Infinite-Width Neural Networks https://arxiv.org/abs/2011.14522 Yang, Hu et al. (202…
For any reward function and policy , consider the entropy-regularized reward Taking as our objective the (expected, discounted…
Note: these are personal notes, taken as I was refreshing myself on this material. They're mostly stream of consciousness and probably not…
A function is measurable with respect to [ sigma-algebra ]s on its domain and on its range if the pre-image of any event is…
The core insight that got me interested: "moments of recognizing your thoughts drifting and bringing them back to your breath" are not…
Apr 24, 2023 be clear on what you're doing in each session. don't mix concentration and emptiness I guess this is related to Tucker's…
Dec 13, 2023 Sangha session with Dustin: Something I realized during a concentration practice is that being honest about how ‘well’ the…
Generally I think the dharma is deeply true and that [ meditation ] done right is healthy and potentially very beneficial. But I struggle…
Specific states or abilities that can arise from skillful meditation: feeling [ equanimity ] first cessation seeing nimitta accessing…
To train a [ transformer ] layer on a sequence of length requires the output of the attention computation where are matrices and is…
Suppose we want to do [ automatic differentiation ] on a [ computational graph ] of sequential length . This could equally well be a…
Described, among other places, in Unlocking the Emotional Brain . Insofar as much of Buddhism is about dissolving [ samskara ]s…
One last thought mental models are so, so important. When I think about computer modeling. It's actually great computers are powerful they…
Like democracy , meritocracy is the worst form of social organization, except for all the others that have been tried. Of course it is good…
References: Risks from Learned Optimization in Advanced Machine Learning Systems A [ reinforcement learning ] algorithm attempts to find the…
Chemically, a substituted [ phenethylamine ]. Like [ dopamine ] but with methyl groups hanging from the two oxygens, and another oxygen…
Generally this means training some aspect of the learning procedure itself. There is then an inner-loop learning procedure, which follows…
Unlike most modern [ deep learning ] systems, humans: don't have separate training/test phases (though we may have wake/[ sleep ]) don't…
Stuart Russell told the story of giving a talk on meta-reasoning at Stanford, with Don Knuth in the audience, where he opened with a slide…
n-methyl-[ amphetamine ]
Pali (Buddhist) term for [ loving-kindness ]
From @visakanv on Twitter: (relevant to [ nothing matters ])
A lesson from [ Tucker Peck ]: unresolved questions are the worst thing in meditation. For example, you're just sitting down to practice…
Considering a bilevel optimization problem (or saddle point problem) on the two-argument function , in general it holds that That is, the…
Short descriptions of things, when they exist, must capture some kind of structure. The principle of [ Occam's razor ] posits that we should…
Mirror descent is a framework for optimization algorithms: many algorithms can be framed as mirror descent, and proofs about mirror descent…
What pieces of [ mirror descent ] can we automate? See also [ natural gradient implementations ] Given a mirror function , we can compute…
(originally from 2020-04-29) On another note, last night I tried to dictate (on Otter) my sense of my life goals. I came up with a very…
[ Otter notes ]: Can I explain what a mixed effects model is from a graphical model standpoint? On the inference side, I think it's just…
A mixture-of-experts model consists of a set of functions , the 'experts', and a gating function that determines how to select which…
I have a [ strong opinion weakly held ] that doesn't seem to be wildly shared in the [ approximate Bayesian inference ] community: reverse…
Original paper: Finn, Abbeel, and Levine, ICML 2017, https://arxiv.org/abs/1703.03400 An approach for [ meta learning ] that works with any…
Often we don't explicitly use 'model-based RL' methods, instead people in robotics talk about Sim2Real: adapting a policy pretrained in a…
Stack: goal: sample from conformations of arbitrary hydrocarbons (or whatever). simpler goal: sample from conformations of ethane. simpler…
Naively you might think that the government just decides how many dollars there should be, and that's that. This is not true. Since [ IOUs…
A monoamine oxidase (MAO) is an enzyme that breaks down mono-[ amine ] neurotransmitters such as [ dopamine ], [ serotonin…
A very natural form of [ meta-reasoning ] that selects the most promising computations. The simplest form of 'expanding' a node assumes a…
There is a connection between moral realism and belief in [ qualia ]. If you see "experience" ([ awareness ]) as a real, fundamental aspect…
I don't hold the moral view that it's better to be a morning person than an evening person. Having always tended towards a later sleep…
In any human-to-human interaction, language carries some very important high-order bits, but it can only carry a few bits. It can help…
This is one of the big problems with the world. Not the only one, and not the only way to look at it. But it's everywhere. status : a…
(see David Graeber https://www.strike.coop/bullshit-jobs/ ) Most work is oriented towards achieving [ instrumental goal ]s. But most…
I like this take on working with procrastination from a [ nondual ] [ awareness ] perspective: From the viewpoint of "the beyond…
maneuvering: the bike goes where I look. look around the turn I want to do. keep elbows up. shift body weight to counterbalance the bike. E…
possible refs: google's multimodal architectures: https://webcache.googleusercontent.com/search?q=cache:https://towardsdatascience.com…
From a conversation I had about [ attention ] mechanisms in deep architectures. Maybe that terminology is too suggestive --- it's just a…
We say that a random vector is multivariate Gaussian with mean and covariance matrix if it can be written where is a vector if i.i.d…
[ thoughts on multivariate causalimpact ]
This was originally a section of breakup.org, written several years ago. this is more related to jobs and identity, but for cases when I get…
I want to intentionally spend my time well. I remember back in grad school I would spend evenings reading papers, just as a form of growth…
I've identified as a 'tech' person, but I now feel uncomfortable in many tech circles. What is tech and what does it mean to be a tech…
It's a useful exercise to occasionally reflect on what I value. stab 1: Generally pro tech, creating new things, non-zero-sum contributions…
Recommended by Michael Edward Johnson:
A 'natural' abstraction is one that we expect any agent (or at least, a wide range of agents) to develop because it gets at something…
We don't typically think of it this way, but you can derive a [ gradient descent ] step as finding the point that minimizes a linearized…
How can we automate [ natural gradient ]? See also [ mirror descent implementations ]
Cool trick: some applications can improve on nearest-neighbor lookup by training 'Exemplar SVM's. Instead of matching against a set of…
My position (a [ strong opinion weakly held ]) is that global utility is currently negative, and probably always has been. It's conceivable…
A negligible function is a function such that, for any positive integer there exists an integer such that for all , i.e., that…
Christian Naesseth, Fredrik Lindsten, Thomas Schon (2015): http://proceedings.mlr.press/v37/naesseth15.html The main idea: In an SMC…
Like the proverbial half-full glass, smart people can look at the same reality of the current capacities of neural nets, and come to…
Sometimes you'll see people say that neural nets 'just' memorize and interpolate their training data. No one denies that neural nets with…
Parts of a neuron: dendrites: these branch out to receive connections from other cells axons: these branch out to send signals to other…
The folklore no-free-lunch 'theorem' in machine learning says that, for any pair of learning algorithms, there exists some dataset on which…
No-self is one of the [ three characteristics ] that traditional Buddhism holds are present in all phenomena. In later Buddhism, the…
https://arxiv.org/abs/1712.02390 Basic idea: optimizers like Adam and RMSProp already keep track of posterior curvature estimates. These are…
Instead of directly targeting a specific rate of inflation, a [ central bank ] may target a fixed rate of nominal GDP growth, which is equal…
One way to model real-world [ causality ] is a bunch of forces working with and against each other. In this view, no individual force…
NFTs 101: https://medium.com/@intenex/nfts-101-why-nfts-are-a-generational-innovation-4626ae803e3b Among many other things, NFTs are…
Obligatory disclaimer: there will never be a drug to turn you into Einstein. Most of effective high-level thinking lies in 'software…
References: Gu et al., Continuous Deep Q-Learning with Model-based Acceleration (2016). Instead of modeling directly, we build a network…
Something can be true but not 'true enough'. That is, you have a compelling causal theory for why X should increase Y. It might be that the…
I've started reading The Art of Doing Science and Engineering by Richard Hamming. History of computing: Analog computing goes back forever…
Because: [ goals are arbitrary ]: achieving a goal, or failing to, doesn't really matter because the goal was arbitrary anyway. From the…
There's a spiritual idea, in Buddhism and elsewhere, that there is "nothing to do": everything is already suffused with "primordial…
Don't invert that matrix: https://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/ Seven sins of numerical linear algebra…
A very incomplete and maybe nonsensical intuition I want to explore. Classically, people talk about very simple [ reward ] functions like…
A few (relatively uninformed) thoughts about on- vs off-policy [ reinforcement learning ]. Advantages of on-policy learning: On-policy…
Original: Daily reflections What am I grateful for today?:: Some goals : Goals for the next ~year:: Goals for the next ~month:: Goals for…
The brain doesn't have separate models of each of the [ sense gate ]s (and thought). Instead it just stores each moment of perception as a…
Informally, a function is a one-way function if it is easy to compute but hard to invert. Or more generally, hard to pseudo-invert, i.e…
These are things that I might plausibly decide I want to work on when I sit down on the weekend. Expanding nodes on this graph. Blogging…
How do we maintain values when our models of the world shift? If someone's goal in life is to "do God's will", and then they come to believe…
As Josh Marshall said , at the beginning of the Trump presidency: "Optimism is not primarily a prediction but an ethic, a philosophy, a way…
If is a [ martingale ] and is a [ stopping time ], then any of the following conditions implies that : The stopping time is bounded…
Ken McLeod claims that 'emotional reactivity' is the origin of suffering. Pain consists both in what happens and in our reaction to it. But…
mnemonic: OIL RIG = 'oxidation is losing (electrons), reduction is gaining (electrons)' in contrast to [ acid-base chemistry ], which is…
This is how [ mitochondria ] produce most of their [ ATP ]. Mitochondria have an outer membrane and an inner membrane, so there are two…
Look again at that dot. That's here. That's home. That's us. On it everyone you love, everyone you know, everyone you ever heard of, every…
References for PDEs: commutant's Youtube videos: https://www.youtube.com/playlist?list=PLF6061160B55B0203 Fundamental PDEs wave equation…
Basic notes from https://www.stats.ox.ac.uk/~doucet/andrieu_doucet_holenstein_PMCMC.pdf Setup: we have parameters and time series model…
Chocolate tasting: buy a bunch of high-end, single-origin chocolate bars. Parcel them out blind. Give people a pad to take notes on what…
We often see optimization problems with objectives of the form where is the main function of interest (e.g., training loss in machine…
“Remember that a person’s name is to that person the sweetest and most important sound in any language.” Dale Carnegie (How to Win Friends…
When you're thinking about doing something that feels right to you, it's easy to get caught up in worrying about what other people will…
reading the perceiver papers from Deepmind: Perceiver: Jaegle et al 2021 https://arxiv.org/abs/2103.03206 Perceiver-IO: Jaegle et al 202…
In the [ 5-MeO-DMT ] trip where I experienced [ ego death ], I saw a [ magical display ] of beautiful colors and flowing motion and…
The AI Effect refers to the widely-recognized phenomenon that 'once we know how to do it, it's not AI'. For example, playing chess well…
I always found it weird that philosophy spends so much time talking about specific historical philosophers. Who cares what Aristotle, or…
When considering one's impact on the world, it's important (? or at least tempting) to think about about your value-over-replacement. If you…
(see also: [ large models ]) There's a viewpoint that neural nets just memorize the training data, so the more training data you have, the…
Developed and widely used in Russia, phenibut is an analogue of [ GABA ] with a phenyl ring substituted at the carbon, giving it the name…
Why Nature Chose Phosphates (science.org)
To His Coy Mistress Andrew Marvell, 1681 Had we but world enough and time, This coyness, lady, were no crime. We would sit down, and think…
The paradoxical thing about pointing-out style meditation teaching is that you can't really explain the instructions when they're unclear…
(see also my [ deep RL notes ] from John Schulman's class several years ago, which cover much of the same material) We can approach…
There are a few ways to do this. Google's PaLM uses rotary embeddings so it seems like that's probably close to the state of the art? But…
Different experimental conditions may give rise to different outcomes . For example, let the variable indicate whether a person is…
Prayer is a form of [ therapy ]. It's about clarifying your values: figuring out what you really want so that you can ask God for it. and…
A [ stochastic process ] is predictable if its value at time is fully determined by information available at time . Any fully…
A really valuable exercise that I should consider building into my routine is to regularly try to make and write down explicit predictions…
Consider an agent that is purely concerned with [ predictive processing ]: finding the optimal [ compression ], or equivalently the optimal…
The theory of predictive processing seems to be attracting a lot of interest in neuroscience and [ meditation ] circles. I want to try to…
https://www.quora.com/What-is-a-preference-cascade A lot of how people act is driven by how they think they're 'supposed' to act. There's…
AI / RL Distributional RL book: https://www.distributional-rl.org/ Alignment Sequences: Value learning: https://www.alignmentforum.org/s…
A Bayesian view of (one aspect of) [ attention ] inspired by a conversation with Shamil Chandaria on [ predictive processing ]. (but this…
It seems like there is, or can be, a virtuous relationship between privacy and generalization. You don't want to memorize too many…
Illegible privilege We often talk about the 'privilege' associated with certain categories: being born white, straight, male, rich, in a…
(I got this concept from SuccessfulFriend.) As people grow up and form their identities, they need models, and not just models; they need…
Can we think about [ generative flow network ]s as a potentially tractable formulation of probabilistic program induction?! executing a line…
Many [ probabilistic programming ] researchers frame their work as part of the broader problem of [ artificial intelligence ]. Artificial…
A short note on interpreting a transformer layer as performing maximum-likelihood inference in a Gaussian mixture model: https://arxiv.org…
Matt Levine explains how a financier might react to losing a billion dollars: Sure sure the risks didn’t work out but you probably have a…
A probability space consists of: A set of outcomes aka possible worlds; these represent all the ways the world might be. This is the…
(aka, why frequentists will always make more money) In the "real" (corporate/governmental) world, most high-level decision making is…