What happened to genetic algorithms?

Eight years ago in March of 2017, evolutionary algorithms seemed on track to become the AI paradigm, before being supplanted by the LLMs that we all know and love (tolerate?). OpenAI proposed that evolutionary strategies could replace–or at least supplement–reinforcement learning: they are simple to implement and scale well. Since these optimizers are population-based, they’re parallelizable and make minimal assumptions. Three months later, however, Google released “Attention is All You Need” and the transformer was born. It took OpenAI roughly a year to then develop GPT1 (and we know the rest).

For those unfamiliar, genetic algorithms fall within a category of optimization procedures called metaheuristics. Acting analogously to evolution, they simulate populations of candidate solutions, select and retain the best, and modify the survivors for the next generation. By design, these algorithms lack closed-form solutions and strong guarantees. That’s their whole gimmick, and it’s a double-edged sword. They can solve black-box problems with practically no assumptions, but guarantees for finding optimal solutions are limited, and convergence speeds are slow and only known in specific contexts. 

Also, the true umbrella term is not actually genetic algorithms but “evolutionary computation” (EC), comprising four historically distinct subfields (though the schools have blended together in recent years):

  1. Genetic algorithms (the most commonly known), 
  2. Evolutionary strategies,
  3. Evolutionary programming, and
  4. Genetic programming. 

Given this is an applied statistics blog, what’s the relevance? It turns out that model selection (or as ML practitioners call it, hyperparameter tuning), can be framed as an optimization problem. Instead of doing grid search, LASSO, or old-fashioned forward selection, we can also search the space of models using evolutionary algorithms. In difficult-to-traverse, discrete model spaces, that could mean the difference between success and no attempt at all. 

What about beyond model selection? From the perspective of expanding models continuously, any unmodeled selection event (even if done with clever metaheuristics) still artificially shrinks posterior intervals and inflates confidence. Not an issue since evolutionary algorithms can also be used for general model estimation! That opens the door to optimizing model parameters over complicated non-differentiable geometries. Technically, we could even try EC instead of OLS for linear models (although I don’t know why we would want to).

So why is it that they have fallen in disfavor? Are they in disfavor?

Thomas Bäck, a prominent researcher in EC, (and collaborators) reviewed the last thirty years of developments. They claim the main issues that have prevented adoption are:

  1. A “bestiary” of nature-inspired algorithms lacking fundamental grounding has made the field incohesive, and practitioners confused
  2. There is no unifying paradigm for designing algorithms, and benchmarks only apply to specific problems, making specific implementations idiosyncratic and hard to evaluate

On the other hand, we shouldn’t undersell developments like CMA-ES (covariance matrix adaptation evolutionary strategy–a bit of a mouthful). In simple terms, you adapt a shared covariance matrix to shape, scale, and correlate your Gaussian perturbations based on current best candidates and past generations. This shared strategy is in contrast to previous algorithms where each individual updated its own parameters. It represents the current state of the art, ranks highly on numerical benchmarks, and has had success in combustion feedback control, multi-cellular migration, and ultrasound imaging. But then again, the core algorithm hasn’t changed in the thirty years since its inception, at least according to Bäck.

So, there may be a more fundamental issue, for which we can speculate. Evolving things takes lots of computation, and when I say lots, I truly mean lots.

A paper by Yampolskiy called “Why We Do Not Evolve Software” has made a compelling case for why evolutionary algorithms aren’t a cure all. The following are incredibly rough estimates and make various assumptions about total global biomass, cell replication rates, and what constitutes biological computation, but essentially: the biosphere that evolved us can compute ~10^40+ FLOPS (floating point operations) per second. It also took that biosphere billions of years to evolve us. All our supercomputers on Earth combined can only compute ~10^22 FLOPS per second. This would seem to imply that the cleverness of an efficient genetic search has to cut through ten to thirty orders of magnitude! That’s a hefty requirement. Not impossible, but probably impractical in many cases.

What does this mean about the future of genetic algorithms? It seems that solutions will somehow need to bypass the combinatorial explosion of the solution space, since getting the compute required without that would take hundreds of years, even with an ever-constant Moore’s law. One direction may be to continue exploring the interplay between local and global solutions (like in CMA-ES and particle swarm optimization). The most concrete step forward would be improving benchmarks and validation. Maybe a unifying paradigm for automatically designing evolutionary algorithms could then develop, similar to what we have in probabilistic modeling.

Simple pseudocode for transformer decoding a la GPT

A number of people have asked me for this, so I’m posting it here:

[Edit: revised original draft (twice) to fix log density context variable and width of multi-head attention values.]

This is a short note that provides complete and relatively simple pseudocode for the neural network architecture behind the current crop of large language models (LLMs), the generative pretrained transformers (GPT). These are based on the notion of (multi-head) attention, followed by feedforward neural networks, stacked in a deep architecture.

I simplified the pseudocode compared to things like Karpathy’s nanoGPT repository in Python (great, but it’s tensorized and batched PyTorch code for GPU efficiency) or Hunter and Phuong’s pseudocode, which is more general and covers encoding and multiple different architectures. I also start from scratch with the basic notions of tokenization and language modeling.

I include the pseudocode for evaluating the objective function for training and the pseudocode for generating responses. The initial presentation uses single-head attention to make the attention stage clearer, with a note afterward with pseudocode to generalize to multi-head attention.

I also include references to other basic presentations, including Daniel Lee’s version coded in Stan.

If this is confusing or you think I got a detail wrong, please let me know—I want to make this as clear and correct (w.r.t. GPT-2) as possible.

ChatGPT (4) can do 3-digit multiplication

I’ve heard a lot about GPT’s inability to do long multiplication. Most people had either tried giving it a few examples (i.e., “few shot learning”) or giving it a few examples and asking it to explain its work (i.e., “chain of thought”). I thought I’d just try to teach it the way you might teach a person. Or if you want to get fancy, I use few shot learning of chain of thought (a very common technique to get it to do this kind of problem). The bottom line: I could get it to do 3-digit multiplications but not 4-digit. The title originally continued “but not 4-digit multiplication,” but I realized I haven’t shown that—just that my prompt doesn’t do it.

Bob

BC: Now we are going to do multiplication.  As a refresher, here is a list
of all one-digit multiplications.

0 * 0 = 0
0 * 1 = 0
...
[removed for blog]
...
9 * 8 = 72
9 * 9 = 81

To multiply a two-digit number by a one-digit number, we break the two
digit number apart into its tens place and its one place and add them,
and then distribute through to get simpler additions, and then
multiply,

31 * 2
  = (30 + 1) * 2 
  = (30 * 2) + (1 * 2)
  = 60 + 2

If a number ends in zeros, we multiply the digits before the zero then
add the zeros back in at the end.  So to do 30 * 2 we multiply 3 * 2
to get 6 and then add the zero back to get 60.  Another way to think
of it is that 30 * 2 = 10 * 3 * 2 = 10 * 6 = 60, where we just wind up
multiplying by the tens place.

If we have a bigger number, it'll have more digits in the final
addition, as in

85 * 9
  = (80 + 5) * 9
  = (80 * 9) + (5 * 9)
  = 720 + 45
  = 765

To multiply two digit numbers times each other, we break them both
down, multiply all the factors, then add the resulting terms
together. 

23 * 45
  = (20 + 3) * (40 + 5)
  = (20 * 40) + (3 * 40) + (20 * 5) + (3 * 5)
  = 800 + 120 + 100 + 15
  = 920 + 100 + 15
  = 1020 + 15
  = 1035

We handle products of three digit numbers the same way.

234 * 567
  = (200 + 30 + 4) * (500 + 60 + 7)
  = (200 * 500) + (200 * 60) + (200 * 7) + (30 * 500) + (30 * 60) + (30 * 7) + (4 * 500) + (4 * 60) + (4 * 7)
  = 100000 + 12000 + 1400 + 15000 + 1800 + 210 + 2000 + 240 + 28
  = 112000 + 1400 + 15000 + 1800 + 210 + 2000 + 240 + 28
  = 113400 + 15000 + 1800 + 210 + 2000 + 240 + 28
  = 128400 + 1800 + 210 + 2000 + 240 + 28
  = 130200 + 210 + 2000 + 240 + 28
  = 130410 + 2000 + 240 + 28
  = 132410 + 240 + 28
  = 132650 + 28
  = 132678

ChatGPT (4, Aug 3)

Certainly! To multiply 12 by 3, we can follow the method you outlined.

First, we break 12 down into its tens and ones places:

12 = 10 + 2

Next, we distribute 3 through the sum:

(10 + 2) * 3 = (10 * 3) + (2 * 3)
= 30 + 6
= 36

So 12 * 3 = 36.

Three digits works

Bob

That's really fantastic.  Now we want to see how far you can take this.  Can you do a three digit product?  How about trying

423 * 567
  = 

ChatGPT (4, Aug 3)

Certainly! To multiply 423 by 567, we'll follow the same process: breaking down each number into its individual place values (hundreds, tens, ones), and then using distributive multiplication.

First, the breakdown:
  423 = 400 + 20 + 3
  567 = 500 + 60 + 7

Now, the multiplication:

(400 + 20 + 3) * (500 + 60 + 7) 
 = (400 * 500) + (400 * 60) + (400 * 7) + (20 * 500) + (20 * 60) + (20 * 7) + (3 * 500) + (3 * 60) + (3 * 7) 
 = 200000 + 24000 + 2800 + 10000 + 1200 + 140 + 1500 + 180 + 21 
 = 200000 + 24000 + 2800 + 10000 + 1200 + 140 + 1500 + 180 + 21 
 = 224000 + 2800 + 10000 + 1200 + 140 + 1500 + 180 + 21 
 = 226800 + 10000 + 1200 + 140 + 1500 + 180 + 21 
 = 236800 + 1200 + 140 + 1500 + 180 + 21 
 = 238000 + 140 + 1500 + 180 + 21 
 = 238140 + 1500 + 180 + 21 
 = 239640 + 180 + 21 
 = 239820 + 21 
 = 239841 

So 423 * 567 = 239841.

Generalizing to 4 x 2 digits

It can generalize to different numbers of digits.

Bob

How about a 4 digit number times a 2 digit number?

2357 * 15
  = 

ChatGPT (4, Aug 3)

Certainly! To multiply 2357 by 15, you can use the same distributive multiplication process.

First, break down each number into its place values:
 2357 = 2000 + 300 + 50 + 7 
 15 = 10 + 5 

Now, perform the multiplication:
 (2000 + 300 + 50 + 7) * (10 + 5) 
 = (2000 * 10) + (2000 * 5) + (300 * 10) + (300 * 5) + (50 * 10) + (50 * 5) + (7 * 10) + (7 * 5) 
 = 20000 + 10000 + 3000 + 1500 + 500 + 250 + 70 + 35 
 = 30000 + 3000 + 1500 + 500 + 250 + 70 + 35 
 = 33000 + 1500 + 500 + 250 + 70 + 35 
 = 34500 + 500 + 250 + 70 + 35 
 = 35000 + 250 + 70 + 35 
 = 35250 + 70 + 35 
 = 35320 + 35 
 = 35355 

So  2357 * 15 = 35355.

4 digits fails

That’s nice. Unfortunately, 4-digit multiplication goes off the rails. It generalizes well to breaking the numbers down and calculating the intermediate terms, but it can’t handle adding together 16 numbers. Maybe another style of teaching it multiplication would work, like breaking the problem down into each place of one of the numbers and then adding at the end like people do. This seems to be something that could be taught in the reinforcment learning stage of ChatGPT training.

Maybe next time. People have shown various things in this problem: (a) it helps to break the numbers apart with spaces (it’s bad at tokenizing numbers by default), (b) writing numbers with ones place first helps, and (c) training a transformer directly to do arithmetic with numerical coding works better still. Feel free to add more in the comments.

But how about addition

OK, this is backward, but I didn’t want to make this too tedious. I started with addition and it’s good at that, too. Here’s the prompt if you’re curious.

Bob

You are an expert at doing arithmetic by hand.  I am going to
remind you of the protocol to follow so that you can show your
work and teach others.  Here are the rules for adding single digits:

0 + 0 = 0
0 + 1 = 1
...
[snip]
...
9 + 8 = 17
9 + 9 = 18

To add two-digit numbers, we add all of their parts together, e.g.,

62 + 3
  = 60 + 2 + 3
  = 60 + 5
  = 65

First, we reduce the two-digit arithmetic using the rule 2 + 3 = 5.
Then we reduce 60 + 5 = 65 because when the two-digit number ends in
0, we just concatenate the one-digit number.  Here's another example.

86 + 7
  = 80 + 6 + 7 = 80 + 13
  = 80 + 10 + 3
  = 90 + 3
  = 93

To add two two digit numbers that end in zero, just add their
first digits and add a 0 to the result, e.g.,

20 + 30 = 50   (because 2 + 3 = 5, followed by 0)
90 + 40 = 130  (because 9 + 4 = 13, followed by 0)

I would like you to do some two digit addition for me and show your
work.

To add two digit numbers, we just continue to break them apart.

12 + 43
  = 10 + 2 + 40 + 3
  = 10 + 40 + 2 + 3
  = 50 + 5
  = 55

Here, we had to break both numbers apart, 12 = 10 + 2, and 43 = 40 +
3.  We then rearranged the order of terms so that we could reduce 10 +
40 = 50 and 2 + 3 = 5.  Then we just added the result.  Another
example

59 + 82
  = 50 + 9 + 80 + 2
  = 50 + 80 + 9 + 2
  = 130 + 9 + 2
  = 130 + 11
  = 100 + 30 + 10 + 1
  = 100 + 40 + 1
  = 100 + 41
  = 141

We always reduce the smallest terms first and then break apart again
if we need to.  In the final step, we use the rule that if the last
two digits of a three-digit number are zero, we can add to a two-digit
number by concatenation.

Now let's see if you can do some.  I want you to answer in the form
above with a sequence of equalities until you get to a single number
for the answer.

12 + 3
  = 

It generalizes nicely from two to three digit examples.

Bob


And now three digit numbers,

123 + 45
  =

ChatGPT

Certainly, let's add 123 and 45 by breaking them down into their hundreds, tens, and ones components and proceed step-by-step:

123 + 45
= 100 + 20 + 3 + 40 + 5
= 100 + 20 + 40 + 3 + 5
= 100 + 60 + 8
= 100 + 60 + 8
= 160 + 8
= 168

I started by breaking apart both 123 and 45 into their respective components. For 123, the components are 100, 20, and 3; for 45, the components are 40 and 5. Next, I rearranged the terms to make it easier to add the tens together (20 + 40 = 60) and the ones together (3 + 5 = 8). After that, I combined the hundreds, tens, and ones sums together: first combining 100 and 60 to get 160, and then adding the remaining 8 to arrive at 168.

Report on the large language model meeting at Berkeley

I’m not the only one who thinks GPT-4 is awesome. I just got back from an intense week of meetings at the Large language models and transformers workshop at Berkeley’s Simons Institute for the Theory of Computing. Thanks to Umesh Vazirani for organizing and running it so calmly.

Here are the videos of the talks.

Human feedback models

I gave a short talk Friday afternoon on models of data annotation.

  • Bob Carpenter. Softening human feedback improves classifier calibration

  • The step from the language model GPT-4 to the chatbot ChatGPT involves soliciting human feedback to rank potential outputs. This is typically done by converting the human feedback to a “gold” standard and retraining the baseline GPT-4 neural network.

    Chris Manning (who introduced me to statistical natural language processing when we were both professors at CMU), provides a nice high-level overview of how OpenAI uses reinforcement learning with human feedback to try to align ChatGPT to the goals of being helpful, harmless, and truthful.

    Chris Manning. Towards reliable use of large language models: better detection, consistency, and instruction-tuning.

    Humans rank potential ChatGPT output and their feedback is used as input for a Bradley-Terry model of conversational reward. This is then used to retrain the network. Chris suggests a much simpler approach than the one they use.

    While at the workshop, John Thickstun, a Stanford CS postdoc, pointed me to the following (and also filled me in on a bunch of technical details in between sessions).

    Chen Cheng, Hilal Asi, and John Duchi. 2022. How many labelers do you
    have? A close look at gold-standard labels
    . arXiv.

    It makes some simplifying assumptions to prove results including the bias of majority voting. I show similar things through simulation in a case study I posted on the Stan forums a while back,

    Bob Carpenter. For probabilistic prediction, full Bayes is better than point estimators.

    More on that soon when Seong Han and I finish our recent paper on annotation models.

    LLMs and copyright

    The highlight of the entire event for me was a captivating talk by a brilliant professor of intellectual property law at Berkeley

    Pamela Samuelson. Large language models meet copyright law.

    If you’re at all interested in copyright and AI, you should watch this. She very clearly explains what copyright is and how the law sees works of artistic expression different than function and hence how it sees code and (other) artistic works differently. She also covers the basis for the cases currently being litigated. She was also masterly at handling the rather unruly crowd. I’ve never been to an event with so many interruptions by the audience members. It was almost like the audience was practicing to be DARPA program managers (a notoriously interruption-prone group).

    Is ChatGPT just a stochastic parrot?

    The other talk I’d encourage everyone to watch is

    Steven Piantadosi. Meaning in the age of large language models.

    He goes over a lot of the cognitive science and philosophy of language necessary to understand why ChatGPT is not just a “stochastic parrot.” He focuses on the work of, wait for it…Susan Gelman (Andrew’s sister). Susan works in my favorite area of cognitive science—concept development.

    I can’t recommend this one highly enough, and I’ll be curious what people get out of it. This one’s closest to my own background (my Ph.D. is joint in cognitive science/computer science and I taught semantics, philosophy of language, and psycholinguistics as well as NLP at CMU), so I’m curious how understandable it’ll be to people who haven’t studied a lot of cognitive anthropology, philosophy of mind, and cognitive development.

    Sanjeev Arora gave a talk about combining skills and how he did a simple combinatorial experiment of combining five “skills” among thousands of skills (not defining what a skill was drove the audience into a frenzy of interruptions that’s quite something to watch). This is behavior that “emerged” in GPT-4 that isn’t so great in the less powerful models.

    Speaking of parrots, the West Coast Stats Views blog (which Andrew often cites) is parroting mainstream chatbot FUD (fear, uncertainty, and doubt); see, e.g., Thursday tweets. I say “parrot” because the blog’s Thursday posts just point to things we used to call “tweets” before a certain someone decided to throw away a brand name that’s become a verb. The irony, of course, is that they accuse GPT of being a parrot!

    Scaling laws

    There were a couple of nice talks by Yasaman Bahri (DeepMind) on Understanding the origins and taxonomy of neural scaling laws and Sasha Rush (Cornell/Hugging Face) on Scaling data-constrained language models. These are important as they’re what allows you to decide how much data to use and how large a model to build for your compute budget. It’s what gave companies the incentive to invest hundreds of millions of dollars in infrastructure to fit these models. Sasha’s talk also discusses the roles researchers can take who don’t have access to big-tech compute power.

    Watermarking LLMs

    Scott Aaronson (UT Austin, on sabbatical at OpenAI) gave a really interesting talk,

    Scott Aaronson. Watermarking of large language models

    The talks a masterclass in distilling a problem to math, explaining why it’s difficult, and considering several solutions and their implications. I felt smarter after watching this one.

    You might also want to check out the competition from John Thickstun, Robust distortion-free watermarks for language models, which takes an encryption key-based approach.

    In-context learning

    “In-context learning” is what people call an LLM’s ability to be given zero or more examples and then to complete the pattern. For example, if I say “translate to French. oui: “, we get what’s called “zero-shot learning”. If I give it an example, then it’s called “one-shot learning”, for example, “translate to French. notre: our, oui: “, and so on. ChatGPT can manage all sorts of nuanced language tasks given only a few examples. It’s so good that it’s competitive with most custom solutions to these problems.

    Everyone kept pointing out in-context learning did not learn in the sense of updating model weights. Of course it doesn’t. That’s because it’s conditioning, not learning. The whole process is Markovian, returning a draw from Pr[continuation | context]. The issue is whether you can do a good job of this prediction without being AI-compete (i.e, a general artificial intelligence, whatever that means).

    A whole lot of attention was given to ChatGPT’s poor performance on arithmetic problems coded as characters like “123 * 987”, with a couple different talks taking different approaches. One trained a transformer with the actual digits and showed it could be made to do this, pointing to the problem being the encoding of math in language. Perhaps the most insightful is that if you give GPT access to an API (with in-context learning, no less), it can call on that API to do arithmetic and the problem goes away. The final talk on this was during the lightning sessions, where Nayoung Lee (a Ph.D. student from Wisconsin-Madison) showed if you reversed the digits in the output (so that they were least significant first, the way we usually do arithmetic), transformers could be trained to do arithmetic very well; here’s a link to her arXiv paper, Teaching arithmetic to small transformers.

    Sparks of artificial general intelligence

    Yin Tat Lee kicked off the program talking about the Microsoft paper, Sparks of general AI. If you haven’t read the paper it’s a fascinating list of things that ChatGPT can do with relatively simple prompting.

    One of the interesting aspects of Yin Tat’s talk is his description of how they treated ChatGPT (4) as an evolving black box. To me, this and a bunch of these other talks that people did probing GPT’s abilities, point out that we need much better evaluation methods.

    Federated learning and task specialization

    For those interested in hierarchical modeling and the idea of a foundation model that can be adapted to different tasks, Colin Raffel (UNC/Hugging Face) gave an interesting talk on federated learning and deployment.

    Colin Raffel. Build an ecosystem, not a monolith

    This was not unrelated to Sasha’s talk (perhaps not surprising as they’re both affiliated with Hugging Face). They also talk about the ecosystem of image models sprouting up around Stable Diffusion and the ability to fine-tune them using low rank methods.

    OpenAI is closed

    Ilya Sutskever, CTO of OpenAI, gave a talk that I can best describe as adversarial. It was the only talk that filled the room to staning room only. He said he couldn’t talk about anything computational or anything related to LLMs, so he spent an hour talking about the duality between probability and compression and Kolmogorov complexity.

    Large language model alignment “bias” and cultural consensus theory

    The way contemporary chatbots like ChatGPT work is by “aligning” a “foundation” model. I think cultural consensus theory (a statistical model, not a contentious issue for school boards) can provide a model for the sociology of alignment.

    The foundation: attention models

    In a nutshell, language modeling is the simple task of predicting the next subword (“called a token”) based on the previous sequence of subwords. The state-of-the-art had stalled for years on n-gram models that use the previous n subwords (usually with n < 5). In 2017, a team of Google researchers released a paper titled "Attention is all you need," which introduced the current state-of-the-art neural network architecture for language modeling. The breakthrough was in extending the context length into the thousands (GPT 3.5 uses 4K, GPT 4 has 8K and 32K models) with an attention model that figured out which parts of the context to concentrate on. The fundamental bottleneck is that computation is quadratic in context length (though it's all on GPU, so that's a massive numbers of flops for relatively low power).

    The 2017 paper introduced the so-called “transformer” architecture, which combines multiple attention “heads” in parallel. The original application was to translation, but it’s the self attention component that was extracted for use in LLMs. The “T” in “GPT” is for “transformers” (the “GP” is for “generative pretrained”). What researchers have found is that the heads learn different aspects of prediction, such as different syntactic structures, much like any mixture model.

    There’s a beautiful two-hour YouTube tutorial by Andrej Karpathy that builds up the entire transformer architecture piece by piece in a Colab notebook you can also use. Karpathy applies it to building a Shakespearean chatbot. It assumes you know Python, but is otherwise quite gentle, starting with an intro to n-gram language models and softmax.

    Garbage-in, garbage-out

    The current crop of large language models have been trained on vast amounts of human text, primarily collected through the internet. As you might imagine, including sources like Reddit and 4chan and Twitter leads to a broad set of what can most charitably be called “points of view.” Even on technical issues, the web is cluttered with material that should probably not be the basis for serious work—homework exercises for intro data science classes clutter GitHub and StackOverflow, every statistician and their cousin’s experimental code seems to be wrapped up as an R package, scripts from ancient versions of software persist, etc.

    Alignment: from LLMs to chatbots

    After building these powerful, transformer-based large language models (LLMs), people realized that they were really good at generating text. As in they blew away any previous compression record (just like the TV show Silicon Valley!). You can convert a language model into a compression scheme using prediction by partial matching (PPM) with arithmetic coding, the reference implementation of which was designed and coded by Radford Neal (with Ian Witten and John Cleary) in 1987. Seriously, they should win an ACM/Turing award just for the quantum leap in text compression.

    The early LLMs could write computer programs, translate Pascal to Fortran and Swahili to English, and generate new recipes given only a list of ingredients or new episodes of TV shows. But they tend to ramble off topic, tend to “hallucinate” (the term of art for when LLMs make things up; it’s called “dreaming” for diffusion models like Midjourney), and tend to be fluid with the points of view they find in training data. They’re just as happy telling you how to make a bomb in your basement and where to set it off as they are telling you how to make a soufflé in your kitchen and how to serve it. And if you “jailbreak” the current ChatGPT, it’ll still be happy to tell you how to try all sorts of dangerous, illegal, and morally and ethically questionable activities.

    OpenAI’s approach to preventing the LLMs from spewing dangerous and/or toxic garbage is to fine tune the large language models with human feedback (HF) using reinforcement learning (RL, and together RLHF). Their stated goal was to “align” the language models to be (a) helpful, (b) truthful, and (c) harmless. While this sounds like an objective task presented this way, the notions of truthful and harmless are difficult to pin down and require subjective judgement calls. Even helpfulness is a slippery notion in that help that’s too verbose or specific isn’t helpful. What one person takes to be self evident in these realms can be considered lunacy by others.

    OpenAI either implicitly or explicitly chose the point of view of a West-coast American liberal, which is the far left of the mainstream US political spectrum, even though it’s relatively conservative by European standards. They could’ve just as easily decided to give ChatGPT the perspective of the far right of the mainstream US political spectrum and it would’ve had a very different perspective and a different segment of the population would be complaining about its biases.

    Cultural consensus theory

    In 1979, Phil Dawid and David Skene introduced a statistical model of crowdsourcing for medical records. The idea is that there’s a latent true value of something like whether a patient smokes, and doctors looking at medical records are going to give you a noisy measurement of that value. The same kind of model can be applied to radiology and doctors classifying medical images for stage of cancer, etc. Or to NeurIPS paper reviews. The model assigns accuracies and biases (too positive or too negative) to the raters and infers the underlying rating most consistent with the ratings (given the rater accuracies and biases).

    David and Skene’s model was independently rediscovered by many, including by me with the help of Andrew and Jennifer Hill (it was my gateway model into Bayes and there’s an example of how to code it in the Stan User’s Guide). As Andrew tells me, no matter what model you look at, a psychometrician probably introduced it 50 years ago (e.g., Elo is just a rescaled Bradley-Terry model, which is from 1950).

    In 1986, A. Kimball Romney, Susan Weller, and William Batchelder published “Culture as consensus: a theory of culture and informant accuracy”, which introduced cultural consensus theory (CCT). It shouldn’t be surprising that it was published in an anthropology journal, because anthropology is cross-cultural sociology. Batchelder and Romney later published a paper, “Test theory without an answer key” in Biometrika; think IRT 0PL model but with unknown true answer, which is the Dawid and Skene model.

    The twist that CCT introduced to take it beyond David and Skene’s model was a mixture model for the “truth.” That is, they assumed there might not actually be a single consensus point of view among raters. This would be a good idea for crowdsourcing, too, where the respondents are often a mix of spammers and people making a good-faith effort (it’s really more of a continuum).

    I think it would be interesting to apply CCT to ChatGPT. It’s the same kind of thing that folks do in applying ideal point models to voting.