Lauren Coffey reports:
A team of researchers at the University of Colorado at Boulder and the University of Pennsylvania have created AI tools to help admissions officers by analyzing students’ application essays.
The tools help admissions officers identify seven key traits in essays, including teamwork, perseverance, intrinsic motivation and willingness to help others. The researchers published their study in October . . .
The endgame is clear. Students use the computer to write the essays, then the university uses the computer to evaluate the essays. No need for any human in the loop.
Same thing with internet advertising. The AI writes the ad and conducts the bidding to put the ad online. The ad is then read by various bots so that the company that sells the ad gets to say they have traffic. We human users fight through the muck to find actual content.
I guess this will happen with movies too. The AI writes the movies, which are then streamed, and then it’s watched by bots who review them and other bots who use them as raw material for the new movies they write. Again, no human needed.
This came up when Jessica and I wanted to get our play Recursion performed at Neurips. At first the organizers were very supportive, but then they said no because “the play itself is not written or performed by AI.” You can’t fight the system!
” Students use the computer to write the essays, then the university uses the computer to evaluate the essays. No need for any human in the loop.”
There was a scene in the movie Real Genius where the students at (movie) Caltech were not showing up to lecture, just leaving recording devices in their seats instead. Eventually no students showed up and neither did the teacher. The class was just the student’s devices recording the teacher’s recording.
More seriously, many things about universities (such as admissions essays) were always silly. If this reductio ad technology absurdum exposes that, so much the better.
I do hope there is a bit of full circle with these ideas back to more traditional statistics concepts (for lack of a better term). A colleague is working on a qualitatively similar project (not admissions, but looking at text and getting out numerical scores of how well something was performed). How much is the machine assessment biased relative to a human? How do we set up process control system to audit and make sure the machine is not going off the rails?
I do not know long term how the essay assessment stuff is going to pan out though. I am not sure a good replacement signal for that.
I think I’m less annoyed by this than you, Andrew, because the broader issue is so infuriating. College admissions essays select for florid or dishonest students, with some weaker correlation for traits like “teamwork, perseverance,” etc. The devaluing of objective (even if imperfect) measures like standardized test scores relative to subjective, severely biased measures like essays has contributed considerably to the muddying of the aims of higher education. It has also helped generate many frustrated, disillusioned students. Perhaps replacing the essay evaluation with AI will make the essays’ inherent fraudulent nature more obvious. College admissions people, however, are highly resistant to logic, fairness, or transparency.
Raghu –
The devaluing of objective (even if imperfect) measures like standardized test scores relative to subjective, severely biased measures like essays has contributed considerably to the muddying of the aims of higher education
I think the aims of higher education should primarily be to help all students achieve their goals, to help all students be better “executives” of their own learning processes.
Norm-referenced (as opposed to criterion-referenced) standardized testing does little to advance those goals. In general, norm-referenced standardized testing advances the obsolete goals of directing students toward a passive approach to evaluating their own learning, and towards an educational paradigm that prioritizes rule-following perpetuating the status quo of existing class structures.
In the very least, there’s no need to apply a zero sum framework to this question.
Helping all students achieve their goals is a great aim, which is *best served* by accurately assessing what skills/knowledge/abilities students have. Ignoring this information serves no one. I am curious if you have ever taught an intro-level / gen-ed course at a large public university. I have, and the fallout of current trends is obvious. My intro physics class contains people whose basic algebra skills — to say nothing of elementary trigonometry — are abysmal, as well as people for whom these are second nature or trivial. Having them all in the same class helps no one. Actually taking seriously the notion of helping all students, promoting active engagement, and stimulating students requires actual assessment. The well-meaning antipathy towards testing *hurts* students, which administrators and armchair instructors, or those who only teach upper-level / highly engaged students, fail to understand.
Sorry for going a bit off-thread, but this does tie back to the original post. Moreover, having read, for example, essays for grad school admissions, I can quite confidently state that these essays’ impressions of perseverance, grit, or motivation have little correlation with what students’ actual perseverance, grit, or motivation are.
While I’m sure Joshua can speak for himself, and probably will, I think you (Raghu) have misunderstood his remark. The key here is the parenthetical remark “(as opposed to criterion-referenced).” I don’t think he has antipathy towards testing as a whole, just towards norm based testing. Norm-based testing, in fact, _is not useful for_ things like matching students to courses based on their existing levels of knowledge or skills. Only criterion-referenced tests can do that.
Moving slightly from defending Joshua’s remark to my own position, I think there should be an “industry” that develops standardized criterion-referenced tests of the knowledge and skills expected as prerequisites for, and intended to be learned in all widely used college-level courses. Those could be used as pre-tests for placement into classes of more or less homogeneous levels of readiness for the courses. I would also want to see these standardized tests used for summative evaluation, replacing the dubious homebrew tests made up by the teachers themselves. Evidently this approach cannot be applied to higher level courses where the expectations and content are idiosyncratic, and the number of potential examinees too small to even carry out useful psychometric analysis of the tests. But for general education courses and “101” level courses in nearly all subjects, I believe this would work well.
Raghu –
Clyde hit on pretty much what I would say in response. I’ll add a some more (of course).
Norm-referenced testing has a distinct advantage: it’s more pedagocally useful. It can inform students (importantly) as well as instructors what material has and hasn’t been mastered. This can help to engage students as active executives of their own learning process. That’s a key step in helping students understand the role of metacognition in learning. Norm-referenced testing reinforces the practice for students of passing over (to tests and instructors) the task of evaluating what is and isn’t known, leading them to be less metacognitive in their approach.
What is the value of evaluating students based on an underlying mechanism of ranking them on how well they perform on tests relative to other students (as opposed to an underlying mechanism of evaluating what they do and don’t know)? I’m not saying you don’t get any useful information from norm-referenced testing. You may even indirectly get information about what students do and don’t know, but it is an inherently inefficient way of doing so. You say as an instructor you want to have information about what students do and don’t know so you can be efficient in addressing students’ knowledge gaps. Why not be more efficient in that process?
But further, I have worked with many intentional students across many different educational contexts. Many were selected for enrollment based based on norm-referenced standardized testing. Sometimes the students from some countries had their scores on norm-refenced standardized testing effectively discounted because they had mastered techniques for doing well on that testing even though that skill did not similarly translate into advanced performance on the course material; as an example, say in business programs where networking and being a self-starter and taking concepts and past experiences and applying them to new, real world contexts was critical.
I’m not saying to just eliminate norm-refenced testing and replace them with essays as the single metrics for evaluating student applications. I’m saying to recognize that norm-referenced testing has real limitations. It has problems both in terms of reliability and validity, or how well it measures what you’re intending it to measure.
“In general, norm-referenced standardized testing advances…an educational paradigm that prioritizes rule-following perpetuating the status quo of existing class structures.”
Standardized testing measures knowledge and problem-solving skill. Nothing more, nothing less. In doing so, it can be sensibly used as an aproximate index for some combination of intelligence, knowledge and dedication.
To say that knowledge and problem-solving skills somehow “perpetuates class structures” is absurd. Quite to the contrary, knowledge and problem-solving skills, to whatever degree any individual has them, perpetuate well-being for *all* humans, even those who’s skills are less, because they benefit from the productivity of those who’s skills are greater. The promote the elimination, not the creation, of rigid class structures.
That’s why, today, even a poor man can obtain a delicious fresh apple any day of the week in much of the world. And that’s why, contrary to the relentless and reckless claims of impending global disaster, more humans are thriving today than ever, by a very large margin. Your efforts to undermine knowldge by attacking standarized testing are entirely designed to perpetuate your own ascension in the resulting debilitating class structure and to dramatically undermine human well-being in the process.
Standardised testing, as done by multi-choice questions, measures a small subset of knowledge because there are only certain ways you can ask questions in that paradigm. For example, in multi-choice questions you can’t ask. “Give me two examples of ways to calculate the amount of fuel needed to get to the moon. Outline your assumptions for each case. Critique each way and then give reasons for deciding which option you prefer.” My impression of multi-choice tests is that all knowledge gets reduced to definitions.
And while it can test for innate “problem solving” skills, it can’t differentiate between that and “memorised all the methods of solving problems that could be encountered on this test”. And money can definitely help to make learning the latter more efficient and even fun, depending on the tutor.
Anonymous –
To say that knowledge and problem-solving skills somehow “perpetuates class structures” is absurd. Quite to the contrary, knowledge and problem-solving skills, to whatever degree any individual has them, perpetuate well-being for *all* humans, even those who’s skills are less, because they benefit from the productivity of those who’s skills are greater.
First, Just look at the simple fact that (in this country) you can largely predict performance on norm-referenced standardized testing (n-rst) based on class status. Unless you’re willing to assume that class status is in itself a measure of knowledge and “problem solving skills” there’s a problem with your absolutist paradigm.
The[y] promote the elimination, not the creation, of rigid class structures.
I would say that’s somewhat more true in a country like China, where students from very poor regions can gain access to more “elite” schools by virtue of performance on n-rst. In the US, however, not so much. We need only to look at the numbers on class mobility to evaluate this question. Children from higher SES backgrounds have all manner of structural advantages for performing well on n-rst, most obviously access to expensive test prep support.
And that’s why, contrary to the relentless and reckless claims of impending global disaster, more humans are thriving today than ever, by a very large margin.
Hmmm. Maybe you’re extending this discussion just a tad into domains that are somewhat tenuously linked?
Your efforts to undermine knowldge by attacking standarized testing are entirely designed to perpetuate your own ascension in the resulting debilitating class structure and to dramatically undermine human well-being in the process.
Lol. I’ve spent many years working to help students overcome class barriers. Independent of the questions related to the value of n-rst relative to college entrance essays, I certainly think your deciphering my “design” to “undermine knowledge” by discussing issues on a blog might be a tad overwrought. I may not be quite as Machiavellian as you seem to think.
This is a process that’s been ongoing for about a century. It was prompted, according to Malcolm Gladwell, when the alumni of Princeton were horrified that 20% of the class (of 1922, I think) was Jewish. The criteria, then, were changed away from formally quantification standards to more qualitative, being “well-rounded”. Scott Aaronson, on his blog, writes about it:
https://scottaaronson.blog/?p=6
In short, the college system is meant to be gamed. To address your post, this is simply part of the process.
I think this counterexample is likely to be more the case. Last week, a Phd student at the university of Minnesota was kicked out of the program for using ChatGPT on his qualifying exam. So, the exam was taken remotely. In this video, the evidence against him is presented. He’s also interviewed, in which he explains why his answers are exactly as those from ChatGPT. IMO, the evidence is strong (against him). Nonetheless, this isn’t a process of quantification. It’s still just a guess that he cheated; and because it’s just a guess, it seems the harsh penalty of expulsion is wrong and unfair.
The link: https://www.youtube.com/watch?v=MNonKtRrw7Q&t=220s
I watched the vid at the link. The thing is that it’s not just about how much the student’s work matches chatGPT but how much the student’s and chatGPT’s work match the allowed texts. If the questions of the test are quite precise then both the student and chatGPT could pull out the same text from the allowed text books. It doesn’t mean the student used chatGPT but that they were both able to get out of the text book the same required answers.
If other students have used chatGPT and “optimised” it for the same questions then the output from chatGPT could look very much like text book answers. The more technical the subject, the more words there will be in common because of the precision of word meanings needed.
Agree.
They should change the exam questions to be about applying the knowledge to a novel situation rather than repeating back definitions.
I can see this career being easily replaced by bots if that is the criteria for a PhD.
I’m don’t know the American system but I thought this test was about “do you have the background knowledge before embarking on a PhD”. The PhD is where you make advances based on the accumulated knowledge, perhaps even in a novel situation.
I have to agree with Anoneuoid here. If the test is to see if someone has the background appropriate to pursue a PhD, then I’d say the test is worthless. Don’t we want to know if someone is capable of finding the background knowledge? Who cares is they possess it? Too much testing is asking people whether they have answers in their heads – a skill perhaps relevant to a role as an expert witness on the stand. Far more important (and growing in importance every day) is whether they can ask good questions and whether they can find good answers – whether it is from a book, a person, or an AI, or from their own head. My reaction to almost all concerns about AI in education is that we need to ask better questions – then “cheating” becomes almost irrelevant.
The PhD preliminary exam should be timed and proctored and handwritten. Not remote. All exams should be like this, honestly.
Also, his advisor supports him, I don’t think the evidence against him is very strong. Automated plagiarism checkers routinely give very large similarity percentages even when the work is original. Generating Chat GPT logs over and over to compare to his answers is even worse. He also says he used Chat GPT only to check his English, not his answers.
I also agree with mpledger and Anoneuoid.
Why time limited? Does that have anything to do with competence or potential? I don’t think so – the only rationale I can think of regards cheating. And, frankly, with 50 years of teaching now under my belt I find cheating not worth my time, of no interest, and the least valuable (as in =0) part of teaching. Before you think that I allow students to cheat without any consideration, my protection is to ask questions/assignments that are virtually cheating-proof. And I’m not worried about them using resources: books, people, online searches, LLMs, online forums, etc. If they can get help and find effective answers, I don’t care much whether it came from inside their head on a time-limited exam or whether they had the capacity and creativity to get help with a good answer.
I mostly use LLMs as a help with my writing and transforming my writing to different modes (e.g. reducing the word count, a layperson’s summary, a media release). I would be very wary of using it for knowledge unless I had a really good command of the subject area. My colleague asked chatGPT for a literature review of some issue and it made up incredibly plausible but non-existent citations of journal articles.
My understanding is that the purpose of the particular exam that the young man took was about whether he had a command of the literature before starting the research aspect of his PhD.
Even if the questions revolved around something novel, a candidate would generally want to approach it from a basis/the definitions of what the accepted literature says about the topic before launching into the novelty of it. With the precision of language necessary for definitions etc there are only so many ways you can say the same thing. It’s the kind of exam that you would need to be able to use LLMs knowledgeably i.e. is the LLM talking truth or rubbish.
Strikes me as a strange partnership. The acceptance rate for the University of Colorado is 81% while UPenn is 6.5%. The UPenn tuition is about double. UPenn should be partnering with Columbia.
I do not know about UPenn partnering with Columbia, but a far more juicy topic is UPenn partnering with Donald Trump:
https://www.phillymag.com/news/2019/09/14/donald-trump-at-wharton-university-of-pennsylvania/
https://www.forbes.com/sites/christopherrim/2019/02/28/heres-why-donald-trump-doesnt-want-anyone-to-know-his-grades-or-sat-scores/
As an European I think you think the human made criteria for application essays is the worst part: ” teamwork, perseverance, intrinsic motivation and willingness to help others.” People of all personalities should get in. Measure the intellect and ability, not “being a nice guy.”
No modern Wittgenstein, Newton, Einstein, Paul Dirac, Teller, gets in if the applicant must be easygoing team players.
Are you implying the schools “Wittgenstein, Newton, Einstein, Paul Dirac, Teller” went to, when they went to them, had fewer “are you a team player” requirements than today?
As I recall, Wittgenstein’s family was quite wealthy. I understand that applicants with this attribute often do well in college admissions today—even if they are not good team players.
Telling your personal history and hobbies and having interviews to check you out before entering college is very American thing.
I’m not familiar with every case, but But in in most places in Europe and Asia admission exams have always been: “This is material you must be familiar with, there is exam at this date, those who score best get in.” or admission based on grades. No person to person interviews, or personal history unless you are applying to psycology, art or something special.
Einstein got into Swiss Federal Polytechnic School trough entrance exam, Wittgenstein the entrance exam for the k.u.k. Realschule in Linz, and so on.
Having interview for admissions and asking personal questions seems just perverse. Maybe that’s a cultural thing.
Sure it’s perverse, but that wasn’t the question. The question was did European universities used to place more de facto ‘conformity’ type requirements on students and professors than American university do today?
It seems as though they did. Didn’t they at one time have to take loyalty oaths to the church of England to a prof at Cambridge?
A good chunk of the students and faculty at my university (a major US one) have a visceral loathing of America. Was that something Prussia put up with in 1860?
Are we going to get to a point where college essays (entrance or otherwise) have to be written spontaneously and in a controlled environment? I recall the GRE process being quite controlled, including cameras. Sounds a tad dystopian, but so does the AI ouroboros.
If you made up “AI ouroboros,” congrats, that’s awesome!
This issue is a well-known AI joke by now. Take a look at these cartoons:
https://marketoonist.com/2023/03/ai-written-ai-read.html
I’ve been wondering for a while when we will reach the milestone where more music is listened to by AI than by humans. We may have already reached it.