(note: this is dancing around the issues around why I think [ probabilistic programming is not AI research ], even if it will be a…
Quit Your Job (palladiummag.com) : Productive exploration requires the application of skilled personal judgment to chasing hunches and…
Exponential Families, Conjugacy, Convexity, and Variational Inference Any parameterized family of probability densities that can be written…
Considering training an [ autoregressive ] model of sequence data (text, audio, action sequences in [ reinforcement learning ], etc.), which…
This note is a scratchpad for investigating the expressivity of the [ transformer ] architecture. In general, one set of intuitions that we…
In [ Dan Brown ]'s telling, these are: mindfulness ( sati ( https://en.wikipedia.org/wiki/Mindfulness (Buddhism))_): paying attention to the…
I grew up in the 2000s reading the New Atheists, where 'faith' was considered a dirty word. Faith was the opposite of reason; it meant…
MACARONI AND CHEESE 4 Tb margarine 5 tb flour Milk 2 tsp mustard 1 garlic clove, minced or 1 tb prepared minced garlic 2 ½ cups sharp…
MACARONI AND CHEESE 4 Tb margarine 5 tb flour Milk 2 tsp mustard 1 garlic clove, minced or 1 tb prepared minced garlic 2 ½ cups sharp…
Ultimately what is attractive in fashion is confidence. you can break almost all of the rules if it's clear that you're doing it as a matter…
I've thought before that fashion is bad because it's about arbitrary trends. But you can also see fashion as good because it's about…
On an evolutionary timescale, it's useful to evolve structures that can learn quickly. The nervous system is an evolved organ system for…
See also Scott Alexander's Ontology Of Psychiatric Conditions: Dynamical Systems - Astral Codex Ten (substack.com) theory of depression.
A filtration is defined by monotonically increasing subsets of a [ probability space ]; that is, subsets such that we have for all…
Yale course: https://oyc.yale.edu/economics/econ-251 MIT course: https://www.youtube.com/playlist?list=PLUl4u3cNGP63B2lDhyKOsImI7FjCf6eDW…
From a Jean Yang tweet : I once attended a talk by the late Nobel Laureate Oliver Smithies where he talked about going into lab on weekends…
Buddists identify five factors as obstacles to [ concentration ] in [ meditation ]: Sensory desire ( kāmacchanda ) Aversion or ill will (…
Interesting and seemingly very powerful perspective on the [ cessation of suffering ]. Most refs on this page are from this twitter thread…
We say that is a fixed point of an update rule if . Update rules can often (though not necessarily) be seen as defining an…
As AGW points out here , it is statistically better to fit a flexible model family, with an inductive bias, than a constrained model family…
Credit to NameRedacted for this refrain https://twitter.com/visakanv/status/1324978566455468035/retweets/with_comments It's a powerful take…
Statements from Forer's experiment : You have a great need for other people to like and admire you. You have a tendency to be critical of…
A key insight of Christianity is that forgiveness is something we do for ourselves : it's not just about extending [ grace ] to the party…
These are the first teaching of the Buddha, after he achieved [ enlightenment ] while [ meditation|meditating ] under a tree. The truths are…
Banks create money by lending. Few understand this. Alice and Bob are on a desert island. Alice has $100, which she deposits in the Desert…
Some drugs, like cocaine, or DMT, come in multiple forms: as some sort of [ salt ] or as a 'free base'. What's the difference between these…
'[ no free lunch theorem ]' arguments are misleading because they consider the space of all possible functions. In fact, we usually care…
[ Andre ] points out the strong free will theorem . This says that electrons sometimes have 'choices': situations where their behavior is…
[ my goals ]
Should I give up having fun in order to do impactful things? It'd be tempting to think that it'd be more virtuous to be serious and mission…
There's often a lot of space between learning [ fundamental ]s and being able to do a thing. Understanding Turing machines didn't…
sources: https://en.wikipedia.org/wiki/Gabapentin https://en.wikipedia.org/wiki/Gabapentinoid https://en.wikipedia.org/wiki/Pregabalin https…
Examples recommended by GPT4: Long Short-Term Memory ([ LSTM ]) : Paper: "Long Short-Term Memory" by Sepp Hochreiter and Jürgen Schmidhuber…
References: Dauphin et al. 2017 https://arxiv.org/abs/1612.08083 Shazeer 2020 https://arxiv.org/abs/2002.05202 https://arxiv.org/abs/240…
Why would you be 'proud' of something you had no control over? The core revelation for me was that pride is the opposite of shame . Most…
Is there such a thing as 'general intelligence'? What capabilities does it require? Is it a goal worth striving for? We usually speak about…
Rules that work in many situations are valuable. If you can cook a burger, you're a McDonald's employee. If you can specify the rules for…
If you need to open a specific lock, you can use a key that encodes the precise information needed to open that lock. If you need to open…
Fundamentally, where does generalization come from? [ causality ]: a model may generalize because it has discovered the true mechanism, or…
Sutton and Barto use this as a general term for any form of interleaving policy evaluation steps with policy improvement steps. This…
Many objects can be generated by a sequence of actions. For example: Generating language by adding one word at a time Generating a molecule…
Small talk can be mindnumbing and pointless. I like the idea of 'big talk, not small talk'. But realistically small talk serves a social…
"What I cannot create, I do not understand". Related to: [ computational complexity ]: provers vs verifiers. [ P != NP ] [ production vs…
Notes on Scott Garrabrant's sequence: https://www.lesswrong.com/s/4hmf7rdfuXDJkxhfg The geometric integral is to products what the…
a Buddhist point. I was trying to fast last night and kept being tempted to relax it slightly . "It won't hurt anything if I just have one…
An intelligent [ agent ] should work to understand the world. This understanding takes the form of a set of relevant [ abstraction ]s, a…
Chemically, glutamate is the [ acid-base chemistry|conjugate base ] of glutamic acid (an [ amino acid ]). It is an [ anion ]; its sodium…
If you fail to achieve your goals, you'll be sad---almost self-evidently. Nonetheless: there is no coherent notion of the 'right' goals to…
Visit [ Andre ] Visit SuccessfulFriend Visit Asian cities Do real bulking and cutting to get in shape Move Learn Mandarin Learn statistical…
see also [ thoughts about kids ] Get them reading young---ideally by age 2 (like Amitav). Learn an instrument from a young age. Let them…
One can never fully deserve grace, but may receive (and extend) it regardless. Probably the most beautiful conceptual contribution of…
Thoughts drawn from my experience doing a CS PhD at a top-4 school around 2010-2016. They may be somewhat applicable to PhDs in other areas…
A recent survey found that ??% of Berkeley grad students suffer from depression. This should be shocking and dismaying. Yet no one seems…
Why do we clip gradients in deep learning? When is it important and what is the right way to do it? It seems like the standard recipe used…
For a normalized distribution , constructed from an (unnormalized) energy with normalizing constant as a function of parameters , in…
How should I think about grades when [ teaching ] a class? I believe in mastery learning. Feedback isn't useful unless students have the…
"How did you go bankrupt? Two ways. Gradually, then suddenly." - Hemingway. It can take a long time to lay the foundations for significant…
A 'graph neural net' is a differentiable, parameterized function whose input or output (or both) is a graph. Discriminative: graph as input…
see also: [ great shows ] Cloud Atlas Pulp Fiction
see also: [ great movies ] Deadwood Sense8
The greek letters used most commonly in finance are probably alpha and beta from the [ single-index model ]. However, the term 'greeks…
When someone we love dies, a sense of possibility leaves the world. Our grief is usually proportional to how much we cared about them…
A nice observation from Percy Liang on the relationship between language modeling and grounded understanding: Just because you don't…
(related: [ communication is processing ]) A big part of growing up is communicating to your future self. Your future self isn't going to…
A thought that just occurred watching Billions. The two deputy attorneys (white guy and black woman ??) are in her apartment, which is nice…
Opposite of a fixed or 'scarcity' mindset. It's important to recognize that the world is nonzero-sum and that improvements are possible. We…
these include things like: posing interesting questions for discussion useful [ generative questions ] like: what have you been reading…
A New Yorker article on [ happiness ]: http://www.newyorker.com/tech/elements/a-better-kind-of-happiness discusses happiness as a source…
Closely related to [ discrete latent variable ]s and to [ reinforcement learning ] with discrete actions. If I do a thing and it goes well…
On Michael Taft's podcast, A. H. Almaas pointed out that an obstacle for most people realizing a sense of divine, nondual love, is some…
Pretend that some other group has published the paper you're imagining: are you excited to read it? Write down ten ideas and ask a mentor to…
The plans I make now include components that would have been impossible for me to conceive of as a kid. At the moment (July 2020), I'm…
What are some things that people are doing now that are just clearly valuable? 3blue1brown : So much of math is hidden behind notation…
[ Dan Brown ] likes to say that 'the view is the meditation'. That is, meditation isn't
People on Reddit worry that there are hundreds of new ML papers every day---how could you possibly keep up? How can you filter the firehose…
[ be open to friendship ] [ act as if you're already friends ] [ people like hearing their name ]
campfires apres ski choir rehearsing under a crisp fall night sky ragnars in vans shabbat various gay gatherings (e.g. pride) -- here the…
[ value alignment ] research often frames the problem as: first, learn the human 'value function' --- for every possible state of the world…
Breville/Polyscience Control Freak: induction burner with temperature sensor. replaces normal stove, electric kettle, maybe sous vide, maybe…
from 2017: remember things I am at least somewhat an expert in, that other people may not have seen: philosophy of bayesian statistics…
I'm sometimes tempted to look back and find patterns in my life, and identify those as "who I really am". For example: maybe I want to be a…
(originally written as a Google doc between 2010-2012) most of this advice is obvious, but still good to remember. students appreciate food…
Multiple senses: An 'ill-conditioned matrix' has a large ratio between its largest and smallest eigenvalue (more generally, see what is a…
References: Gu et al., Continuous Deep Q-Learning with Model-based Acceleration (2016). A technique used in [ model-based rl ], where we…
The Eliezer Yudkowsky school of thought is that immortality is possible, and obviously desirable; any other position is [ learned…
One aspect of depression recently has been feeling like things are pointless, there's nothing valuable for me to do. My entire PhD has been…
[ Shinzen Young ] reframes the traditional Buddhist concept of impermanence as "flow". I think this is starting to make sense to me. I…
Examples: SGD prefers some minima over others
Importance sampling allows us to compute expectations under a distribution using samples from a different distribution , by weighting the…
[ grokking ] / [ phase change hypothesis ] emergence of near-discrete features in large transformers symmetries / non-[ identifiable…
For me, imposter syndrome felt like knowing that there was something deeply wrong with me, something I was missing or didn't get, that would…
things I learned in [ improv ] class there's no need to try to be clever or funny. if you're present and responsive, trust that the…
In the 21st century, humanity is developing a new form of engineering. Rather than manually designing artifacts, we are optimizing over…
Indole alkaloid - Wikipedia Indole - Wikipedia A benzene ring fused with a [ pyrrole ring ]
Ways to specify inductive bias: Feature engineering Prior distribution acts as regularizer in MAP estimates Graphical model (constraint on…
Talia Ringer says that these are one of the most beautiful, foundational ideas in programming languages: https://twitter.com/TaliaRinger…
Around 15min into https://www.youtube.com/watch?v=gWAZFuz_mFc , [ Ram Dass ] claims there are 'infinite doorways'. You start on the…
The Leibniz calculus notation using infinitestimal quantities like or is simultaneously Very sensible and intuitive, but also Constantly…
References: Bae et al. (2022) If Influence Functions are the Answer, Then What is the Question? https://arxiv.org/abs/2209.05364 Grosse et…
To achieve final goals, we have to break them down into a hierarchy of instrumental goals, and then get to work on achieving those. And for…
The [ front-door adjustment ] allows identifying causal affects using a mediating variable that sits on the causal chain between X and Y…
I want to care about an intellectual topic and have friends and colleagues with whom I can enjoy discussing that topic. I used to have…
[ theory of intelligence ]
Boaz Barak writes in GPT as an "Intelligence Forklift." that [ language model ]s seem to function effectively as [ tool AI ] that can…
A lot of discussion around [ artificial intelligence ] implicitly conflates intelligence with [ consciousness ]. It assumes that as we…
Inspired by this tweet: https://twitter.com/davmre/status/841803926051549184 Ways we currently equate intelligence with moral worth: animal…
It's a bit counterintuitive that high interest rates prevent inflation. After all, doesn't a high interest rate mean that the [ central bank…
interesting things dennett says in "darwin's dangerous idea" the "baldwin effect" uses reinforcement learning to construct a version of…
Interfaces enable modularity. In general, standardizing an interface can yield quadratic benefit at linear cost. Suppose we have people…
Things to pay attention to in setting up a new place: Lighting makes a huge difference and can totally change the ambience of a space. Don…
Fasting is a powerful, life-changing idea because it's simple, clear, easy to follow. A diet plan that involves counting calories requires…
References: https://en.wikipedia.org/wiki/Intermolecular_force and linked pages conversations with GPT-4 A molecule is a set of atoms…
The theory of musical intervals almost makes mathematical sense (but not quite): The sensible part: integer ratios The ancient Greeks…
objectives: maximize entropy of the state visitation distribution requires empirical estimates of [ entropy ] maximize mutual information…
David Hume pointed out that there's no logical way to get from 'is' (descriptive) statements to 'ought' (normative) statements. This is…
The isoperimetric problem : among all closed curves in the plane with equal perimeter, which encloses the largest volume? It's well-known…
Update Feb 2025: I now think the existence of [ memory reconsolidation ] makes the point expressed here much less of a blackpill. Factual…
Admitting things that I personally don't know is hard, because it feels like admitting a weakness or failing. But ignorance isn't a…
"it's not my universe": a mantra for [ equanimity ] practice from [ Tucker Peck ]. If you believe in a God who does control the universe…
Apparently Tucker Peck told this story in a talk. Sharon Salzberg says that if she could put one thing on her tombstone it would be this…
Notes copied from Google Docs https://docs.google.com/document/d/1G7Gxo-A3gQrlUx3G4BYmH-GjUWbB7eTOJZcuAs2ii0A/edit most of these things…
From 2017: stay publicly active. work on open side projects, and/or publish, and/or blog. live at such a means that I could lose my job and…
Something I've struggled with: given my position of immense privilege, how can I justify doing wasteful 'fun' things like skiing or clothes…
I don't know if this makes sense, but one intuition I have for karma comes from the observation that the weights of a least-squares linear…
We are given the opportunity to bet some fraction of our wealth on a coin flip with probability . We can repeat this as many times as we…
multiple senses: in machine learning: positive definite (Mercer) kernels in linear algebra: kernel (nullspace) of a linear map in CS systems…
Effects in the brain Ketamine is an antagonist of [ NMDA receptor ]s, and activates [ AMPA receptor ]s. References: Ketamine – Lorien…
pg says: the random things that you learn as a kid make you into a key. Your job is then to find the lock that you fit into. But that's not…
Adults tend to talk to and work with other adults. We don't spend nearly enough time addressing the problems of younger people: college…
I just coined the phrase 'kitchen sink' deep learning for a vague idea that comes to me occasionally. Roughly: rather than using a uniform…
An annoying thing about language-learning tutorials is that they often focus on language that you'll never actually use as a tourist. When I…
Taco Cohen speculates on Large Control Policies as a successor to large language models: https://twitter.com/TacoCohen/status…
Much of statistical practice is concerned with distinguishing signal from noise. For example, significance tests quantify the likelihood…
If you believe that neural nets basically just memorize the training data, then training larger and larger models is hopeless. The…
I don't want to be led. I want to be creative and do things that are dramatically new. Telling other people what to do feels almost evil to…
Happens all the time, at small scales and large scales. Large scale: When I was young, it was possible to imagine becoming a confident…
The set of good approaches is often hidden in an exponentially large space. Learning that an approach is not good doesn't help narrow that…
There's no easy or fast solution for feeling good about leaving academia, because you're giving up some aspect of your [ identity ], and…
(related: [ growing up means becoming wrong ], ask for evidence , [ Seeing Like A State ]) Related to some interactions with [ Josh…
A thing that's tough about going through big personal changes is that it takes a while for your self-model to catch up with your actual self…
With varying degrees of clarity and certainty. We are [ embedded agent ]s. So are any AI systems we build. We exist inside the world; the…
People on the internet have very different standards as to when and how it's okay to ask a question. There are roughly two camps: Search…
https://maskofreason.files.wordpress.com/2011/02/the-library-of-babel-by-jorge-luis-borges.pdf Insights illustrated by this story: Naming…
In principle we can apply [ automatic differentiation ] through any composition of differentiable operations. This lets us get gradients of…
I want to change the world. What does that mean? Suppose I create a billion-dollar company. That's an enormous amount of value. It's many…
tags: created: 2023-12-07 modified: 2023-12-07 References: https://arxiv.org/abs/2006.16236 The usual [ transformer ] [ attention…
A linear time-invariant system is one where the dependence of the output on the input is: linear: an input produces an output , and…
podcasts and audiobooks Alan Watts 'out of your mind' lectures
I've heard it said, and it's been ringing true to me, that the thing to do is live the life you want to have now , not plan to spend years…