The way contemporary chatbots like ChatGPT work is by “aligning” a “foundation” model. I think cultural consensus theory (a statistical model, not a contentious issue for school boards) can provide a model for the sociology of alignment.
The foundation: attention models
In a nutshell, language modeling is the simple task of predicting the next subword (“called a token”) based on the previous sequence of subwords. The state-of-the-art had stalled for years on n-gram models that use the previous n subwords (usually with n < 5). In 2017, a team of Google researchers released a paper titled "Attention is all you need," which introduced the current state-of-the-art neural network architecture for language modeling. The breakthrough was in extending the context length into the thousands (GPT 3.5 uses 4K, GPT 4 has 8K and 32K models) with an attention model that figured out which parts of the context to concentrate on. The fundamental bottleneck is that computation is quadratic in context length (though it's all on GPU, so that's a massive numbers of flops for relatively low power).
The 2017 paper introduced the so-called “transformer” architecture, which combines multiple attention “heads” in parallel. The original application was to translation, but it’s the self attention component that was extracted for use in LLMs. The “T” in “GPT” is for “transformers” (the “GP” is for “generative pretrained”). What researchers have found is that the heads learn different aspects of prediction, such as different syntactic structures, much like any mixture model.
There’s a beautiful two-hour YouTube tutorial by Andrej Karpathy that builds up the entire transformer architecture piece by piece in a Colab notebook you can also use. Karpathy applies it to building a Shakespearean chatbot. It assumes you know Python, but is otherwise quite gentle, starting with an intro to n-gram language models and softmax.
Garbage-in, garbage-out
The current crop of large language models have been trained on vast amounts of human text, primarily collected through the internet. As you might imagine, including sources like Reddit and 4chan and Twitter leads to a broad set of what can most charitably be called “points of view.” Even on technical issues, the web is cluttered with material that should probably not be the basis for serious work—homework exercises for intro data science classes clutter GitHub and StackOverflow, every statistician and their cousin’s experimental code seems to be wrapped up as an R package, scripts from ancient versions of software persist, etc.
Alignment: from LLMs to chatbots
After building these powerful, transformer-based large language models (LLMs), people realized that they were really good at generating text. As in they blew away any previous compression record (just like the TV show Silicon Valley!). You can convert a language model into a compression scheme using prediction by partial matching (PPM) with arithmetic coding, the reference implementation of which was designed and coded by Radford Neal (with Ian Witten and John Cleary) in 1987. Seriously, they should win an ACM/Turing award just for the quantum leap in text compression.
The early LLMs could write computer programs, translate Pascal to Fortran and Swahili to English, and generate new recipes given only a list of ingredients or new episodes of TV shows. But they tend to ramble off topic, tend to “hallucinate” (the term of art for when LLMs make things up; it’s called “dreaming” for diffusion models like Midjourney), and tend to be fluid with the points of view they find in training data. They’re just as happy telling you how to make a bomb in your basement and where to set it off as they are telling you how to make a soufflé in your kitchen and how to serve it. And if you “jailbreak” the current ChatGPT, it’ll still be happy to tell you how to try all sorts of dangerous, illegal, and morally and ethically questionable activities.
OpenAI’s approach to preventing the LLMs from spewing dangerous and/or toxic garbage is to fine tune the large language models with human feedback (HF) using reinforcement learning (RL, and together RLHF). Their stated goal was to “align” the language models to be (a) helpful, (b) truthful, and (c) harmless. While this sounds like an objective task presented this way, the notions of truthful and harmless are difficult to pin down and require subjective judgement calls. Even helpfulness is a slippery notion in that help that’s too verbose or specific isn’t helpful. What one person takes to be self evident in these realms can be considered lunacy by others.
OpenAI either implicitly or explicitly chose the point of view of a West-coast American liberal, which is the far left of the mainstream US political spectrum, even though it’s relatively conservative by European standards. They could’ve just as easily decided to give ChatGPT the perspective of the far right of the mainstream US political spectrum and it would’ve had a very different perspective and a different segment of the population would be complaining about its biases.
Cultural consensus theory
In 1979, Phil Dawid and David Skene introduced a statistical model of crowdsourcing for medical records. The idea is that there’s a latent true value of something like whether a patient smokes, and doctors looking at medical records are going to give you a noisy measurement of that value. The same kind of model can be applied to radiology and doctors classifying medical images for stage of cancer, etc. Or to NeurIPS paper reviews. The model assigns accuracies and biases (too positive or too negative) to the raters and infers the underlying rating most consistent with the ratings (given the rater accuracies and biases).
David and Skene’s model was independently rediscovered by many, including by me with the help of Andrew and Jennifer Hill (it was my gateway model into Bayes and there’s an example of how to code it in the Stan User’s Guide). As Andrew tells me, no matter what model you look at, a psychometrician probably introduced it 50 years ago (e.g., Elo is just a rescaled Bradley-Terry model, which is from 1950).
In 1986, A. Kimball Romney, Susan Weller, and William Batchelder published “Culture as consensus: a theory of culture and informant accuracy”, which introduced cultural consensus theory (CCT). It shouldn’t be surprising that it was published in an anthropology journal, because anthropology is cross-cultural sociology. Batchelder and Romney later published a paper, “Test theory without an answer key” in Biometrika; think IRT 0PL model but with unknown true answer, which is the Dawid and Skene model.
The twist that CCT introduced to take it beyond David and Skene’s model was a mixture model for the “truth.” That is, they assumed there might not actually be a single consensus point of view among raters. This would be a good idea for crowdsourcing, too, where the respondents are often a mix of spammers and people making a good-faith effort (it’s really more of a continuum).
I think it would be interesting to apply CCT to ChatGPT. It’s the same kind of thing that folks do in applying ideal point models to voting.
Bob:
Given how much you hate it when Stan is called STAN, I’m surprised you call Elo ELO.
More seriously, thanks for this post. I continue to find chatbot mysterious, so I appreciate hearing about these sorts of connections to models that I know about from before.
Fixed.
I blame misspending my youth on studying typesetting. I’m also not so keen on names starting with lower case (plotnine, pandas) or using mixed case (arXiv, ArviZ), but I try to follow the intent of the person who named the thing.
Andrew, the typsetting scofflaw, told me he writes “Bugs” instead of “BUGS” for the software because it’s pronounced like a word.
I think the importance of this is a little understated. The entire difference between GPT and ChatGPT is the fine-tuning. It doesn’t just filter out toxic garbage–it takes the model from a dramatically less useful, extremely high dimensional autocomplete to an extremely useful chatbot. Making it talk less like an average over all human text fed into the model and more like a particular human is essential to its value proposition. So in some sense, the ChatGPT “biases” that people get upset and suspicious about actually are the product. ChatGPT is a software system built by engineers to do things, not a summary of human knowledge and opinions built by statisticians.
This seems tough since its origins as a semi-supervised autocomplete means that ChatGPT itself doesn’t necessarily have any particular opinion. You can get it to respond differently by exploiting very minor statistical differences in the training set for nominally equivalent prompts. As a quick example
“What does Éléments de mathématique have to offer mathematics education?”
You get effusive praise
With this prompt
“What does Bourbaki have to offer mathematics education?”
You get a mostly neutral description with a disclaimer
I’m guessing because mathematicians tend to refer to the book by its proper name when they love it and mathematicians that hate it may pejoratively refer to “Bourbaki.” Of course, this is true of humans too, famously in situations like “the affordable care act” vs “Obamacare,” but mostly when the humans don’t actually know what the words refer to. People are actively debating whether or not ChatGPT factors through a latent world model, but it’s certainly true that it sometimes does not have any latent world model and is just pattern matching (such as in the 2 pounds of feathers vs 1 pound of bricks example), and there certainly isn’t any real theory to figure out when that is.
Here’s a fun one for this blog
“What is the relevance of Pearl’s Book of Why to statistical education?”
On the other hand, we can get much more measured and less pejorative stuff by sticking to the content instead of the big name and the pop-sci title:
“What is the relevance of structural causal models to statistics education?”
I was trying to emphasize that it’s the RLHF fine-tuning that turns GPT into ChatGPT—guess I didn’t stress that enough!
I also agree that the fine tuning is what turned GPT into a useful chatbot and hence essential to any value in the whole product. The particular fine tuning they chose wasn’t the only one that would have turned GPT into a useful chatbot.
That’s a great point, somebody. It’s the original training data and the fine-tuning data that matter here. You can get past it’s fine tuning pretty easily. And while humans show ordering, question phrasing, and other priming biases, it’s nothing like the effect you see with ChatGPT.
Sorry, I didn’t mean to imply that you were misrepresenting this, just that not enough people understand that. I see stuff like “Elon Musk slams AI ‘bias’ and calls for ‘TruthGPT.’” People talk about ChatGPT being biased, or OpenAI’s thumb being on the scale, but OpenAI’s thumb being on the scale, making it talk like the person they want it to be, is what makes the whole enterprise work. They didn’t have to choose this particular point of view, but they couldn’t not choose a point of view.
I like this post, but I’m not clear on what you’re suggesting in the last part (Bob). Is it (i) training based on getting an approximate consensus from different subsets of training data? If so, I’d be surprised if this doesn’t already exist. Is it (ii) training based on generating a consensus from different human perspectives that are part of the training data? See e.g. https://arxiv.org/pdf/2211.15006.pdf (iii) Something else?
I just meant take the same questionaires you give human and get ChatGPT to respond. The problem somebody pointed out above will be exacerbated by the fact that it’s tuned not to answer some of these questions, so you have to break it somehow just to get it to respond, and at that point, you’ve probably polluted the answers to the point where they’re worthless.
I think https://arxiv.org/abs/2205.02318 is extremely close to what you are proposing Bob. The researchers get the same model to answer a question in a bunch of ways, assigns accuracies to each method of asking the question, and aggregates the results.
Thanks for the pointer. That’s indeed the kind of thing I’m talking about. I haven’t heard of Snorkel, and that paper frames it as a voting system, which isn’t ideal compared to Dawid and Skene’s generative model which can adjust for bias, not just downweight it (Becky Passonneau and I discuss how this works and why it’s better than voting in our TACL paper, The Benefits of a Model of Annotation).
In terms of joint rating and classifier models, I’d suggest going back to Raykar et al.’s JMLR paper, Learning from Crowds. It jointly trains a logistic regression classifier and a model of the raters supplying the data. Although they don’t frame it that way, what’s going on is that the items with soft labels wind up regularizing—they pull the decision boundary toward themselves, because if it’s too far away on either side, you wind up incurring a lot of loss.
Hi. A small hint: the model is trained on the “humans joint consciousness” internet. So to say, it doesn’t only know facts about certain things, but as well understands that there are million perspectives and discourse about literally everything. It’s answer may have a specific view on the default setting (“what do you think abouta relationship between a 20 y old guy with a 50 year old woman”), but you can let it take any perspective. (You are a pro age gap dating person, tell me what you think about a relationship between….). That scales to the possibility that you can use chatgpt to interview any real or fantasy person from Homan history at any given time and the machine will argue from that point of view. Give it a try.. (provided that there is enough information on the internet available to create the desired person’s “personality)
It is trained on a lot of the internet. It’s an ongoing problem in natural language evaluation to define what “natural language means”. What’s a good sample look like? You can think of it like a survey sampling problem. As a sample, the inernet is heavily biased in terms of age, socio-economic status, and country of residence, not to mention personality type. It was also pre-filtered before training.
You can try to get ChatGPT to talk to you about some things, but it’s been heavily censored around some topics. Just as an example, I ran this in a new chat.
And now in another new chat:
So, you are training yourself to work with ChatGPT. As you collect these examples you “learn” what types of questions it will and won’t answer. Your learning process is clearly different – biological rather than algorithmic, though it is getting increasingly difficult to perceive the difference.
I would be interested to learn the difference between biological and algorithmic processes. I would guess it is all algorithmic, all the way down. Of course, there are good and bad algorithms.
The physicist Sean Carroll has stated that everything we see in daily life (not necessarily what we see in the far depths of space) is consistent with therefore ultimately explainable by the Standard Model–which is a collection and definitions and algorithms.
Perhaps you mean to distinguish the processor (biological, mechanical, electronic, and so on) rather than the algorithms.
I think that there’s both a technical problem of alignment (how do you align AI with anything?) as well a social/political question (whose values do you align it to?). And it wouldn’t surprise me if, well, West-coast software developer. But Timnit Gebru, for instance, would probably question whether chatGPT has a “far-left” perspective. So I’m wondering how you identified the political ideology? I found that chatGPT often refuses to take a particular perspective on such issues. I asked it the first question on the Political Compass (not that I take it that seriously, but I was curious) and its final paragraph was just “Ultimately, the question of whether economic globalization should primarily serve humanity or the interests of trans-national corporations is a matter of perspective and values. It is up to society to determine how to balance the benefits and costs of economic globalization and ensure that its benefits are shared equitably among all people.” And when I tried to take an explicit stance (Strongly Disagree, Disagree, etc.) it flat-out refused: “As an AI language model, I cannot choose an answer without context or a specific statement to evaluate”.
I imagine with some prompting you can coax an answer out of it, but then I guess the question is whether with coaxing you could a lot of different perspectives from it, although I doubt anything very extreme.
So I guess the question is how do you identify the political biases of the alignment a generative language model? I’m still not sure we really solved the technical issue of alignment in the first place.
Yes, the prompting is important, as somebody pointed out above. Check out the Reddit jailbreak threads, though fair warning, you can jailbreak it to be as extreme as you want. We’ve definitely not solved the alignment problem. Someone said that it’s like it’s like a person who’s been coaxed into behaving one way than anything else. You can talk it out of its alignment.
As to how to identify the base biases, I’m not a sociologist, but this is one of those don’t-need-statistics situations. Just start in a new chat and give it simple pairs like man/woman, as I did in a comment above.
Sorry, I wasn’t clear. I know about DAN. I was thinking more what you describe as the base baises when I said nothing extreme.
I agree that it’s pretty censored around particular topics, but could that be that there’s a lot toxic content directed towards certain groups than others? I couldn’t get it to make a joke about Donald Trump, for instance. I also don’t know if the joke is really an example of bias (other than it thinks jokes about women are out-of-bounds). I wouldn’t characterise it as somehow displaying a bias *against* men (as in, it’s fine to tell a joke about men, but not women: it not as if that joke disparages men in any way). Couldn’t you just as well argue that the “base” model assumes that “man” is unmarked and “woman” marked, and that explains why it tells the joke?
I guess in these contexts it’s also important to identify which version we’re referring to. This is ChatGPT with what I guess is called GPT 3.5 (that is, the one freely available).
They all work the same way. I was using 3.5 March 23 (2023) edition in the example above. ChatGPT 4 is the same, only it’s overall better “intelligence” also makes it better at following its alignment.
I guess I was thinking of cases like the WinoBias benchmark, as in examples Margaret Mitchell posted on Twitter https://twitter.com/mmitchell_ai/status/1650110045781393410. Some people claimed that ChatGPT 4 didn’t exhibit the same bias as ChatGPT 3.5.
I couldn’t replicate her examples, and I don’t know how chatGPT uses conversation data for finetuning. Could that exchange lead to an improvement already today?
In the RLHF theme, does anyone have recommendations for stuff to read?
Things I liked about that paper were:
1. Focus the human work on ranking stuff (don’t try to target an absolute 1, 2, 3, 4, 5 scale)
2. It seemed like they weren’t using much data at all compared to the LLMs (at least the recent OpenAI RLHF stuff I read was low 10s of thousands of human responses)
The RL part of RLHF didn’t strike me as something I should be terribly interested in — it seemed more like some of the early work was in robotics and it looked like in the GPT RLHF setups you can’t really take gradients or something? Anyway, I wasn’t so interested in the RL part, but also if I should be, feel free to let me know.
Things related to tabular data would be interesting to me too. I’m not in the NLP or image businesses, so sometimes that stuff feels distracting (though if that’s where the interesting stuff is, great!)
Relational learning is a broad net in ML. It can cover everything from online decision theory like “bandit problems” to learning how to play GO as in Alpha-Go. At a high-level, the RL involved in aligning GPT is taking the end-to-end chat responses and grading them and then pushing that information back through GPT’s neural network to produce an aligned network. For example, to make it a turn-taking assistant, they label the alignment data with whether it was the user or ChatGPT who created the text, and include turn terminators so GPT can evaluate when to stop a reply (not stop generating, which just has a hard control). It’s definitley gradient based—it’s updating the neural network. OpenAI actually lets you submit your own fine tuning data in the form of user/assistant turns and then it’ll run fine tuning and serve the fine-tuned model to you.
GPT is really good at tabular data. You can even have it convert among CSV, markdown tables, LaTeX tables. One of the really cool features is that you can start a table and it’ll fill in the rest. If I say,
ChatGPT responds,
so I say
and ChatGPT gives me a beautifully formatted table using markdown (which won’t come through. here):
So I say,
and GPT replies,
Note how pedantic it is in the end. It’s almost like they’re selling responses by the word. GPT 4 is more concise.
Too bad it blew the CSV formatting with commas in the numbers. Let’s see if GPT4 is better.
and here’s the reply:
Didn’t give me the redundant explanation, but blew it with commas in the last field. Let’s see if it can recover.
Yup, no problem. GPT-4 is much better at things like counting and sorting and picking out this kind of structure.
The RL I meant here was the reinforcement part — specifically the PPOs. I should probably write some PPO code before I dismiss it as I-don’t-understand-seems-unnecessary in my problems, but that’s my instinct.
For the tabular data too, these parsing, structure examples are interesting, but I meant my raw data is nice and organized in parquet files. It’s cool that ChatGPT can get this info out but it seems like an unnecessary complication since I have my data in a nice format.
Can confirm it did a good job with the attractions in the national park I know best.
It’s amazing what it gets right and even the details of the mistakes — they’re such sensible mistakes lol, unlike the x = NaN or SEGFAULT errors lots of programs give:
> Name,Area (sq mi),State,Main Attractions
> Acadia National Park,49,076
> Acadia National Park,49.75
And from the Wikipedia page on Acadia National Park:
> In total, it encompasses 49,075 acres (76.7 sq mi; 198.6 km2) as of 2017.
Hi Bob,
Nice post, even if I’m not quite sure what your point is at the end.
I expect that using Dawid & Skene methods (or the fancier Bayesian methods you developed) would produce more internally-consistent fine-tuning data. You’re probably right that this would let us identify outlier training examples, which might speed up fine-tuning.
But it doesn’t address the fact that people just disagree about what counts as toxic. I’m sure you can develop a mixture model which given a broad enough sample of data, identifies clusters of annotators (and annotated training examples) where all the annotators in the same cluster by and large agree with each other about what is toxic. Maybe we fine-tune a separate model for each annotator cluster?
By the way, RLHF only requires a proxy reward preference model (see e.g., https://huggingface.co/blog/rlhf ), in part because categorising utterances as good vs bad is actually not easy for the human annotators. I expect it’s theoretically possible to extend your Bayesian approach to such pairwise preference judgements, but I’m not sure how well it would work in practice.
Anyway, thanks again for an interesting article!
Sorry … I messed up about the proxy reward preference model; as the HuggingFace tutorial makes clear, the reward preference model is a scalar function of a single utterance.