I’m not the only one who thinks GPT-4 is awesome. I just got back from an intense week of meetings at the Large language models and transformers workshop at Berkeley’s Simons Institute for the Theory of Computing. Thanks to Umesh Vazirani for organizing and running it so calmly.
Here are the videos of the talks.
Human feedback models
I gave a short talk Friday afternoon on models of data annotation.
Bob Carpenter. Softening human feedback improves classifier calibration
The step from the language model GPT-4 to the chatbot ChatGPT involves soliciting human feedback to rank potential outputs. This is typically done by converting the human feedback to a “gold” standard and retraining the baseline GPT-4 neural network.
Chris Manning (who introduced me to statistical natural language processing when we were both professors at CMU), provides a nice high-level overview of how OpenAI uses reinforcement learning with human feedback to try to align ChatGPT to the goals of being helpful, harmless, and truthful.
Chris Manning. Towards reliable use of large language models: better detection, consistency, and instruction-tuning.
Humans rank potential ChatGPT output and their feedback is used as input for a Bradley-Terry model of conversational reward. This is then used to retrain the network. Chris suggests a much simpler approach than the one they use.
While at the workshop, John Thickstun, a Stanford CS postdoc, pointed me to the following (and also filled me in on a bunch of technical details in between sessions).
Chen Cheng, Hilal Asi, and John Duchi. 2022. How many labelers do you
have? A close look at gold-standard labels. arXiv.
It makes some simplifying assumptions to prove results including the bias of majority voting. I show similar things through simulation in a case study I posted on the Stan forums a while back,
Bob Carpenter. For probabilistic prediction, full Bayes is better than point estimators.
More on that soon when Seong Han and I finish our recent paper on annotation models.
LLMs and copyright
The highlight of the entire event for me was a captivating talk by a brilliant professor of intellectual property law at Berkeley
Pamela Samuelson. Large language models meet copyright law.
If you’re at all interested in copyright and AI, you should watch this. She very clearly explains what copyright is and how the law sees works of artistic expression different than function and hence how it sees code and (other) artistic works differently. She also covers the basis for the cases currently being litigated. She was also masterly at handling the rather unruly crowd. I’ve never been to an event with so many interruptions by the audience members. It was almost like the audience was practicing to be DARPA program managers (a notoriously interruption-prone group).
Is ChatGPT just a stochastic parrot?
The other talk I’d encourage everyone to watch is
Steven Piantadosi. Meaning in the age of large language models.
He goes over a lot of the cognitive science and philosophy of language necessary to understand why ChatGPT is not just a “stochastic parrot.” He focuses on the work of, wait for it…Susan Gelman (Andrew’s sister). Susan works in my favorite area of cognitive science—concept development.
I can’t recommend this one highly enough, and I’ll be curious what people get out of it. This one’s closest to my own background (my Ph.D. is joint in cognitive science/computer science and I taught semantics, philosophy of language, and psycholinguistics as well as NLP at CMU), so I’m curious how understandable it’ll be to people who haven’t studied a lot of cognitive anthropology, philosophy of mind, and cognitive development.
Sanjeev Arora gave a talk about combining skills and how he did a simple combinatorial experiment of combining five “skills” among thousands of skills (not defining what a skill was drove the audience into a frenzy of interruptions that’s quite something to watch). This is behavior that “emerged” in GPT-4 that isn’t so great in the less powerful models.
Speaking of parrots, the West Coast Stats Views blog (which Andrew often cites) is parroting mainstream chatbot FUD (fear, uncertainty, and doubt); see, e.g., Thursday tweets. I say “parrot” because the blog’s Thursday posts just point to things we used to call “tweets” before a certain someone decided to throw away a brand name that’s become a verb. The irony, of course, is that they accuse GPT of being a parrot!
Scaling laws
There were a couple of nice talks by Yasaman Bahri (DeepMind) on Understanding the origins and taxonomy of neural scaling laws and Sasha Rush (Cornell/Hugging Face) on Scaling data-constrained language models. These are important as they’re what allows you to decide how much data to use and how large a model to build for your compute budget. It’s what gave companies the incentive to invest hundreds of millions of dollars in infrastructure to fit these models. Sasha’s talk also discusses the roles researchers can take who don’t have access to big-tech compute power.
Watermarking LLMs
Scott Aaronson (UT Austin, on sabbatical at OpenAI) gave a really interesting talk,
Scott Aaronson. Watermarking of large language models
The talks a masterclass in distilling a problem to math, explaining why it’s difficult, and considering several solutions and their implications. I felt smarter after watching this one.
You might also want to check out the competition from John Thickstun, Robust distortion-free watermarks for language models, which takes an encryption key-based approach.
In-context learning
“In-context learning” is what people call an LLM’s ability to be given zero or more examples and then to complete the pattern. For example, if I say “translate to French. oui: “, we get what’s called “zero-shot learning”. If I give it an example, then it’s called “one-shot learning”, for example, “translate to French. notre: our, oui: “, and so on. ChatGPT can manage all sorts of nuanced language tasks given only a few examples. It’s so good that it’s competitive with most custom solutions to these problems.
Everyone kept pointing out in-context learning did not learn in the sense of updating model weights. Of course it doesn’t. That’s because it’s conditioning, not learning. The whole process is Markovian, returning a draw from Pr[continuation | context]. The issue is whether you can do a good job of this prediction without being AI-compete (i.e, a general artificial intelligence, whatever that means).
A whole lot of attention was given to ChatGPT’s poor performance on arithmetic problems coded as characters like “123 * 987”, with a couple different talks taking different approaches. One trained a transformer with the actual digits and showed it could be made to do this, pointing to the problem being the encoding of math in language. Perhaps the most insightful is that if you give GPT access to an API (with in-context learning, no less), it can call on that API to do arithmetic and the problem goes away. The final talk on this was during the lightning sessions, where Nayoung Lee (a Ph.D. student from Wisconsin-Madison) showed if you reversed the digits in the output (so that they were least significant first, the way we usually do arithmetic), transformers could be trained to do arithmetic very well; here’s a link to her arXiv paper, Teaching arithmetic to small transformers.
Sparks of artificial general intelligence
Yin Tat Lee kicked off the program talking about the Microsoft paper, Sparks of general AI. If you haven’t read the paper it’s a fascinating list of things that ChatGPT can do with relatively simple prompting.
One of the interesting aspects of Yin Tat’s talk is his description of how they treated ChatGPT (4) as an evolving black box. To me, this and a bunch of these other talks that people did probing GPT’s abilities, point out that we need much better evaluation methods.
Federated learning and task specialization
For those interested in hierarchical modeling and the idea of a foundation model that can be adapted to different tasks, Colin Raffel (UNC/Hugging Face) gave an interesting talk on federated learning and deployment.
Colin Raffel. Build an ecosystem, not a monolith
This was not unrelated to Sasha’s talk (perhaps not surprising as they’re both affiliated with Hugging Face). They also talk about the ecosystem of image models sprouting up around Stable Diffusion and the ability to fine-tune them using low rank methods.
OpenAI is closed
Ilya Sutskever, CTO of OpenAI, gave a talk that I can best describe as adversarial. It was the only talk that filled the room to staning room only. He said he couldn’t talk about anything computational or anything related to LLMs, so he spent an hour talking about the duality between probability and compression and Kolmogorov complexity.
Awesome stuff! Excited to watch “Meaning in the age of large language models.” I do not have a background in any of those topics, so we’ll see what I have to ask GPT-4 about :)
Even if current copyright laws don’t impede LLMs this area of law deserves more thought. Maybe laws need to change.
Consider that most of the awesome performance of ChatGPT at programming derives from massive training sets accumulated at zero cost such as Stackexchange.
In future, if chatGPT works so well, traffic to sites like Stackexchange or blogs in general would dip drastically. In a sense the very success of chatGPT would kill the human creation of the training sets that it thrives upon.
We need to brainstorm how to prevent this.
Copyright needs to disappear. It’s a terrible drag on society that destroys culture. Imagine if it had existed in the way it does now deep into the past. It’s for sure we wouldn’t have Shakespeare or Dickens or Beethoven or any of it. Copyright would have killed those historical artifacts as surely as copyright holders are trying to kill the Internet Archive for digitizing 100 year old 78RPM records. I could get on board maybe with the original copyright… 7 years and one opt in renewal I think it was. After that everything is public domain.
I agree with you about Copyright in the traditional scenarios. I just think we re-examine all our beliefs and laws when the agent in question is one of these new mega LLMs instead of just a lone artist trying to create derivative work.
As an analogy it is fair game for me to photograph anything in public view with my personal camera however the moment we scale this to massive industrial fleets of cars accumulating Terabytes of personal data, the framework of thinking about the activity changes. And sometimes we need other laws to protect privacy and the fabric of society.
I think it’s really down to saying “yeah, corporations can’t be that big”. Break em up. My estimation is no corporation/company needs to be bigger than 10-20k employees, unless perhaps its a regulated public utility.
Exactly, the use of copyright for individuals or small companies is completely different from that of big corporations, who just distort its original intentions to maximize their financial gain. It’s the same with the concept of “open” or patent law.
Actually I think that if copyright is used as a Luddite hammer to smash the machines the powerful use to oppress worker, that’s perfectly legitimate. Not that it’ll work this time, though.
I am for breaking up the behemoths. But alone that won’t fix the perverse incentives created by LLMs.
They came and swooped up content created by others in a mega efficient assembly line fashion.
My current opinion is that derivative work should indeed be encouraged except when it is these gigantic deployments that threaten independent creation of novel content itself. If LLMs improve the incentive for blogging and many other modes of information dissemination will reduce drastically.
Rahul, suppose we get to the point where LLMs trained on curated science and literature content exceed the abilities of most teachers at the elementary or high school level to correctly explain concepts to kids… maybe even they can draw diagrams using SVG, and such. Wouldn’t it be IMPERATIVE that model becomes widely available? Even create a government sponsored organization to curate and tune that as a public utility? Put the big textbook publishers out of business. Individualized Socratic method teaching…
I disagree basically entirely with your premise and I think the solution is to democratize creation of models as open source public goods.
Rahul, suppose we get to the point where LLMs trained on curated science and literature content exceed the abilities of most teachers at the elementary or high school level to correctly explain concepts to kids… maybe even they can draw diagrams using SVG, and such. Wouldn’t it be IMPERATIVE that model becomes widely available? Even create a government sponsored organization to curate and tune that as a public utility? Put the big textbook publishers out of business. Individualized Socratic method teaching…
I disagree basically entirely with your premise and I think the solution is to democratize creation of models as open source public goods.
Daniel: You misunderstand the purpose of teachers. A teacher’s job is to connect with the students and get them to/help them think for themselves, to get them to look things up in Wikipedia for/by themselves. Not to perfectly desctibe advanced concepts. (It is, of course, also to drill them on the basics.)
Beside, if your teachers aren’t good enough, just raise their salaries and/or hire better teachers. (Just as doubling the minimum wage vastly improves the quality of work of minimum wage workers, it’s a neat feature of capitalism that money actually works as an incentive to work harder/better, and lack of money acts as an incentive to not give a damn.)
You also misunderstand LLMs: the technology, by its very definition, halucinates. It has no mechanism (other than kludges added manually after a problem is found (e.g. Bing used to blithely randomly guess at* the LCM of 76 and 0; now it complains)) to check it’s output with reality**. The LLM technogy doesn’t do reasoning; it just randomly recombines undefined tokens. So unless you are good enough at the field at hand to check the output, you can’t use an LLM (some lawyers tried that, it didn’t end well). Sure, they’re great for filling in templates: “Write me an introduction to a tech paper that…” and you’ll have a rough draft. Same for programming. But they don’t think. That’s your job.
*: Again, what LLMs do is exactly and only randomly guess at.
**: Bing seems to have largely given up on the technology: it uses the LLM to paraphrase the question into a form that works to search Wikipedia (or the web) and then uses the LLM to paraphrase that entry. It’s still smarmy and obnoxious in the extreme, of course. OpenAI seems to remain committed to the religious belief that LLMs are somehow more than just random undefined token recombiners.
David, I don’t deny that humans have important functions, I just think one of them is to actually answer more in-depth questions. A kid who wants to know “how does rattlesnake venom work?” is not going to get a satisfactory answer from the vast majority of actual 5th grade teachers today, and no one is going to be able to afford the teacher who would do a good job (because they’d be a PhD in biochemistry and would be more needed in pharma development than teaching kids). Also, rattlesnake venom is just Thursdays question, on Friday the kid will ask “what is a neutron star?” and you’d need a totally different PhD to answer that question.
The idea that searching wikipedia is better than taking a large language model *trained specifically on biochemistry articles from a variety of sources* and having a conversation with it doesn’t seem right to me.
Note that I am not talking about the technology we have today, I am talking about technology of the not too distant future, where a generic chatbot detects the topic of interest, and then invokes one of thousands of custom models which is actually an expert in the topic of interest having read hundreds of thousands of pages of authoritative textbooks and research articles on the general topic and *not* read a lot of other unrelated crap which might confuse it when it comes to answering questions about biochemistry (or whatever its main topic is).
I’m not claiming such things will replace human teachers, only that they will augment the learning process for children because the children will get access to actual expertise from a computer which can actually pay attention to them unlike a teacher who has 30 kids to divide attention between. The computer can customize responses to specific questions, rather than providing a massive 13 section Wikipedia page on venom in general etc. The teacher’s role will be to teach kids to “look things up” sure, but not by typing a couple keywords into wikipedia search, by formulating reasonable full sentence questions and asking the kids to look at the citations provided and soforth.
I don’t know when or if we’ll get that model, my only point is that when these things are possible, they will be necessary public goods we need to create under funding from govt sources and then give away access to instead of keeping them behind Copyright+Patent paywalls
I think it’s worth keeping min mind that the Twitters, Reddits, and StackOverflows themselves collected massive sites of contributed content without compensating the contributors. Ironically, StackOverflow content is licensed under CC-BY-SA, which they motivate by saying
Fast forward 13 years later and they now say they will charge LLM companies for data. I think the bottom line is that we can’t trust any of these companies. When I contribute content to StackOverflow, I’d prefer it be open, including for LLM training.
Bob Carpenter: “When I contribute content to StackOverflow, I’d prefer it be open, including for LLM training.”
But the terms under which you contribute to StackOverflow (cc-by-sa) require that those using your contributions (1) credit you for them and (2) be willing to share them under the same terms that they received them. Neither of those things is within the power of an LLM.
Of course its possible to credit the responsible parties. “This LLM was trained on the entirety of the text produced by the human race”
I don’t mean that as facetious at all. LLMs have no more responsibility to credit individuals than I do when I say things like the atom is made up of subatomic particles in a nucleus and some electrons surrounding it.
If I write a crime novel I don’t cite Edgar Allen Poe as the origin of the genre, like I need permission to use his idea.
There are specific lists of people who were involved in discovering the atom, like Rutherford who guessed the existence of the nucleus etc but this is common knowledge now and when I say it I don’t cite original research.
Everything a LLM says is “vaguely remembered stuff that’s common knowledge if you just spent the last 2 years reading literally the entirety of the internet.”
It’s not quoting Bob off StackOverflow it’s making stuff up based on its background knowledge of which Bob makes up about one part in a trillion.
I get it that academics are all stuck in the 1600s with fights over who invented Calculus etc. But it’s extremely tiresome the way everyone wants to hold priority like a sword of damoclese over human progress… Ready to drop down and cut off anyone’s head who dares to say stuff without tracking down a citation to justify it.
Meanwhile Tai is reinventing integral calculus and publishing it in Diabetes journals and some diabetes researchers who don’t know calculus are learning calculus because of it… Obviously this is bad because the purpose of research is to claim priority and everlasting credit for the original form of every thought, and not as some naive people might assume, to improve the lives of diabetes patients
https://diabetesjournals.org/care/article/17/2/152/17985/A-Mathematical-Model-for-the-Determination-of
Note also that copyright law (which is horribly broken, but nevertheless the law of the land) has a specific notion of a “derivative work” and “training an LLM on a massive corpus of text and then having it talk about the same stuff that Bob Carpenter once talked about on StackOverflow” doesn’t come within the same universe (and shouldn’t!).
Where copyright comes into play is where specific significant sections of the work are directly derived from the original. If the LLM memorizes a page of text Bob wrote and spits it out whenever you ask it the definition of some word, that could be argued to be a derivative work.
Although an LLM can memorize small quantities of stuff, there’s a simple information theoretic proof that it couldn’t possibly memorize a large fraction of what it read. The weights in the neural-net constitute a tiny fraction of the information content of the training corpus.
Cory Doctorow writes some very useful stuff about how Copyright is not the answer to the world of generative AI https://pluralistic.net/2023/02/09/ai-monkeys-paw/
There are better ways to support creative work than copyright. What is the logic of retroactively increasing the length of copyrights? Clearly, this is just a gift to rich corporations.
Like Don Henley? Paul McCartney? Taylor Swift?
These people are all great evidence that artists social justice hypocrites and liars. All of them gave up the rights to their early music in return for music publishers taking on the risk to promote their music, then when they become famous and wealthy they’re incensed that they don’t have the rights to the music that made them famous and wealthy! Boo hoo.
It’s even more odd that the public in general thinks artists are somehow more deserving of their fortunes than, say, Musk, Bezos or other tech entrepreneurs. Tech entrepreneurs have actually provided something beneficial for their fortunes. CNN breathlessly swooned over Taylor Swift because she recently paid her tour truck drivers a “life changing” bonus. But you can bet tens of thousands of times more people made a life changing amount of money from AMZN, MSFT, META, ORCL…and other tech companies with billionaire founders, and that’s before you count the stock holders.
Thanks for providing that summary and, in particular, highlighting Pamela Samuelson’s talk, which I just watched: it is indeed fantastic.
Thanks for the extensive and interesting report.
We all stand on the shoulders of our teachers and those who developed the tools we use. At some point AI will begin to make significant new contributions. (Already AlphaGo made a new Go move in its tournament with the world champion which astounded the champion and the watching experts, and AlphaFold is the world’s best protein-fold predictor.)
It’s annoying how many times the speaker gets interrupted. I’m only 11min into the Piantadosi talk, and he has already been interrupted three times.
Thanks much, will watch a few of these.
I presume you meant: Translate FROM French…
Another typo, but one that I think is so marvelous that you should keep it in: “staning room only”
I watched “Meaning in the age of large language models.” The point seems to be that you can understand something just by reading about it. I already knew this. (I’m not saying that it is a bad talk.)
But if all you ever did was read about it and everything else, could you understand it?
Yes, I can understand many things just by reading about them. So, I don’t see any reason why an AI couldn’t (not claiming the current ones do).
I think the argument is that you understand many things that you have never seen by reading about them because you have interacted with related things in the real world. For example, you can read about a dragon and imagine it, but the description relates to things that you have already seen/touched/heard/experienced. You’ve already seen a lizard or any kind of animal. Scales of some kind. Wings of some kind. Etc. Even concepts such as ‘justice’ or ‘beauty’ can be felt in a variety of ways and experienced in circumstances. If you had only ever read about *anything at all* without any kind of non-textual experience (including without experiencing yourself as a person), would you be able to understand anything? Isn’t that what an LLM is?
The point of the talk (I believe) is that one can understand things just by reading about them. I already knew this, so did not find the talk revolutionary. If you don’t think it is true, you might be more impressed by the talk (or you might not be convinced). I don’t believe anyone is saying that sense information is not also useful.
Helen Keller never saw or heard things she studied and wrote about. Her inputs came by touch only. LLM’s also have a single input source (currently). I expect that adding visual and auditory input capability could improve them, but don’t know what the tradeoff in processing cost would be. (Humans use a lot of their neurons for visual processing, e.g., our single-lens eyes project images on our retinas upside-down; along with blind spots.).
I would argue that “touch” is not a single input. There are a variety of different receptors that can sense pain, vibration, pressure, and temperature. In addition, more than just touch, someone who is deaf and blind still has all the other sensations of being alive including proprioception, emotions, pain, awareness, etc. All of these inputs can build a world that, although devoid of sound and sight, is much richer than a single input.
FWIW, Helen Keller was 19 months old when she lost sight and hearing. that counts for something.
Those (the replies above) have good points, but don’t prove that Helen Keller never understood any concepts which she had read about (in Braille) but had no other sensory experiences of, such as flight. Where by understanding, I mean could make somewhat useful assessments of them.
As for what she learned in her first 19 months, we know that, for example, baby elephants are not born knowing how to coordinate the roughly 100,000 muscle groups of their trunks, and have more brain neurons than humans, many of which have to be trained to manipulate their trunks, by trial and error. I suspect human babies also spend their first months learning to coordinate muscles and other systems. I myself remember no thoughts before the age of four, and those I do remember at the age were only semi-coherent. I know there is a lot of variation in that, though.
Then there is the ancient philosophical dilemma of whether anyone really understands anything, or whether we are all just contemplating shadows on a wall. All our sensory impressions are ultimately processed as electromagnetic changes in our brains.
In any case, the last estimate I made of GPT-3’s equivalent neuron power was roughly a third of a human cerebral cortex (which itself is a smallish fraction of our roughly 80 billion neurons), which to me is a testament to the ability of dedicated processes to do a lot with a (relative) little. (AlphaGo beat the world champion with less than a mouse’s equivalent brain.)
>but don’t prove that Helen Keller never understood any concepts which she had read about (in Braille) but had no other sensory experiences of, such as flight.
But she *did* have sensory experiences that would help her understand flight. She was standing on the ground! Presumably she could also jump into the air.
It’s hard to think of anything that one can read about that doesn’t relate in some way to something else that one has experienced outside of reading that provides a richer ‘understanding’ of what is being read. People, even who are devoid of many senses, still have many more inputs than 1. LLM’s on the other hand, do have a single input. This would seem to me to make comparing the ‘understanding’ of LLMs to that of people, rather fraught with problems.
Empirical evidence has been brought to bear on this philosophical problem
https://en.wikipedia.org/wiki/Molyneux%27s_problem#Responses
By flight I meant birds flying and planes flying. The ability to move long distances without touching the ground . Without any coherent memory of having seen this, if she read about it she might still gain some comprehension of how it works and why it is useful.
As for blind people given sight and having difficulties relating visual identifications to touch, that seems predictable to me based on the long disuse of their visual cortexes causing parts of them to function poorly and other parts perhaps having been adapted to different uses, It takes time to train neurons and once having been trained for one process it is more difficult to retrain them for something else. According to Wikipedia, authentic cases of feral children (who grew up without human contact until teen-aged or older), once found, have been incapable of learning any language.
Perhaps also the formerly blind and feral people lack motivation to learn difficult new skills which they have gotten along without for long times. For LLM’s, we provide the motivation, in their code.
I should however acknowledge the trap of confusing intelligence with sapience or sentience. According to a neuroscientist whose blog I read, neuroscience has different definitions for these qualities, and GPT has demonstrated that it meets the definition of intelligence (at some level), but not the other two.
I don’t know what are good tests for sapience or sentience. They may be as rare as blind people on pogo sticks.