This is Jessica. We recently wrote about multiverse analysis, which takes the idea of sensitivity analysis to the extreme by multiplexing over every reasonable decision you could have made in analyzing your data. I pointed out that it’s now trivial to turn any paper into a multiverse analysis. Just ask Claude Code to create a notebook replicating the analysis in a paper, then add whatever variations you want. Or let it plan the multiverse too by suggesting some common ablations. Today I want to consider what this newfound ease in testing for robustness might mean for science.
There are a few corollaries of this new reality. One is that it’s now trivial to test reproducibility. No human reviewer may have the patience to deal with your messy codebase, but automated evaluation makes it much easier to detect the failures (at least where it’s possible for data to be made public) and prevent them from moving forward. This can take some curation of fallback plans for common failure cases (see, e.g., this recent paper), but once we endow agents with these skills, we can apply them at previously unknown scale. It’s a short hop from there to multiverse writ large.
Replicability testing at scale is also not far off. For example, you can have it synthesize data with the same structure but variation along some dimension. Before long we may also see agents being given permission to collect new data, for example by running online human subjects studies on Prolific or some other platform. The years of work it used to take to run a large-scale replication project may now translate to one grad student’s summer project.
In short, we should expect the level of scrutiny on papers to change dramatically. Most reviewers are not incentivized to look carefully at materials the authors submit beyond the paper text itself. Even when incentivized, human reviewers have limited time and attention. But AI reviewers can scale evaluation of reproducibility, consistency in claims and evidence, and robustness to perturbing inputs or methods slightly. In the trajectory that led from replication crises revelations and pushback against narrowly constrained subject pools (“WEIRD” science), we might appear to be on the verge of living the open science dream.
The question is, What kind of dream will it be? From a surface reading of the last ten years of science reforms, it could appear to be a win-win situation for science. Authors (whether human-AI combos or fully AI) get the assurance that the work they are producing is robust. The scientific record benefits by converging upon what really stands up to scrutiny, not what sells as a story. There is now the potential for improvements to status quo science that would be hard to conceive of five years ago.
On the other hand, if you think the open science movement has largely overindexed on simple techniques that misconstrue what scientific progress means, and empowered rigor signaling over judgment, welcome to your nightmare. I suspect we will see some perversions of real progress when “science as checklist” becomes policy.
Much will depend on how carefully we steer the kinds of checks we implement, and what to do about the results. Here are a few predictions:
In the short term, acceptance rates will drop
Lots of issues that humans missed will be found by AI. In an optimistic view where automated review uses tools on par with the best that are available today, many (most?) of these issues will be real problems. Even if a human with considerable expertise in the field did go through them one by one, I doubt they would disagree that often. If you have’t tried refine.ink , I recommend checking it out. I started using it shortly after it was released and immediately it became part of the pre-submission routine. It can find even the subtle issues in notation and argumentation.
Standardization of AI checks will incentivize AI in production
The best way to pass automated evals will be to plan and build-in robustness tests throughout the research process. This is the obvious path to acceptance rates recovering. As evaluation gets easier, the demand for it will increase, so that evaluation becomes continuous or “always on.” Checks can be run at every commit, every new experiment, every revised claim.
All of this is already happening to some extent. At least in CS, it now feels risky to send papers out if they are still throwing lots of issues when run through AI evaluations–not just the code, but the entire paper. I tell my students to check their work periodically to catch major issues early.
Papers will be “safer” in certain ways
It will become harder to pass off fragile findings, regardless of how compelling the story may be. “Such-and-such conference is no longer taking risks” is something that people have complained about before AI, as academic communities have matured and acceptance rates lowered. Widespread AI checks could take that sentiment to a new level.
The nature of the new status quo that selecting for “safer” papers creates will depend on what kinds of robustness we prioritize. It will depend on what happens to fuzzier criteria like intellectual risk-taking, or whether a paper opens up new ways of thinking versus purports to resolve uncertainty. What’s the value of telling a good story, one that inspires the (human) imagination, versus presenting a robust (if boring) empirical result or incremental advancement to methods? I expect AI to incentivize the latter unless we explicitly intervene to incentivize fuzzier, human-like aspects of taste.
How big a shift robustness-forward, heavily AI-driven science brings is likely to look different depending on what area you’re in, and what constitutes a novel contribution. In fields where incremental change is the norm, public datasets are commonly used, and combining tricks from prior work has a relatively high probability of success (e.g., machine learning), the change may seem more tolerable. It’s less clear to me how fields like social psychology or sociology, where the story, and how it stands in contrast to our expectations, is paramount will respond.
Nuance will be lost
As I mentioned in the last post, the question that multiverse analysis raises—“What exactly do I conclude on the basis of this glorified robustness analysis?”–becomes more important. This is where lots of nuance could be lost. For example, in writing this post, I gave Claude Code one of my papers, a study that compared human image labeling performance when the participants had access to different presentations of prediction uncertainty. I pointed it to the data files, and prompted it to reproduce the results. I also asked it to extend the results by doing a multiverse to vary key analytic decisions. Since I wanted to see what it associated with terms like “sensitivity analysis” and “multiverse,” I gave it very little specific advice.
Ultimately it produced a bunch of variations on the model specification, varying the inclusion of different kinds of random effects and interactions. It also proposed varying the prior (we’d used Bayesian models). We had a little back and forth–for example, initially it looked only at aggregate effects, rather than distinguishing by treatment arms, though our hypotheses in the paper were specific to data conditions. But overall the process was much, much faster and easier than if I had to do it myself, or ask a grad student.
However—in the end, it summarized the results in exactly the way that theorists warn not to: reporting the frequency of significant results across a set of universes that varied in the covariate structure of the model specification. The problem is that these specifications are not draws from a probability distribution. There is only one data-generating process. Treating model variants as if they form a random sample turns analytic flexibility into an uninterpretable frequency. This is why robustness testing at scale does not guarantee insight: we still have to figure out how to interpret the results, and that is hard.
Given that many of the people working on automated evaluation and AI for science are not metascientists, we could see a lot of nonsensical aggregations of results. Consider, for example, the possibility of synthesizing automated meta-analyses of different empirical literatures. If you think meta-analyses are questionable because it’s not clear what the “average effect” estimated over a heterogenous set of studies even represents, or take issue with the conventional throw-it-all-into-a-simple-random-effects-model approach, brace yourself for a flood of highly precise, poorly defined aggregated estimates.
Policy experiments will be possible on a much shorter time scale
On the other hand, one of the frustrating things about metascience has been that it’s hard to predict how corrective measures will impact science as a whole. Many of the social sciences are more conservative than CS (and haven’t faced the urgency of submission numbers that CS venues are facing), and so policy change to publication practices is slow. The louder and more convincing reformers have profited from this–they can tell a convincing story about how preregistration or other open science methods will save science and win lots of support, without having to prove it. As AI speeds up paper production, and makes it easy to implement new evaluation procedures relatively uniformly at scale, so will our ability to get feedback more quickly on how the published record can shift with various interventions. Metascience stands to become more empirical.
Scientific self-play in the (slightly) longer term
We should expect more and more reliance on AI to figure out the way around the errors. Ultimately we get “self-play,” as John Horton calls it, where different sets of agents propose, implement, stress test, and critique.
This is where things could get fascinating. We know that the inductive biases of large language models, when left to dialogue, can lead them to converge on strange equilibria, like the Claude bliss attractor effect. What happens when the selection pressure is not for fluency, but for reproducibility, transparency, and sensitivity to perturbations? What kind of science emerges when agents are rewarded not for sounding reasonable, but for surviving stress tests? What’s the feel of an equilibrium of low epistemic risk?
It’s probably not going to look much like the science we’ve become accustomed to. Will it be “feels like rigor” on steroids, or will we eventually get genuine innovation and robustness?
Of course, depending on what types of checks become policy, things could get obviously stupid, like when those (human or AI) overseeing algorithmic review decide to filter on null hypothesis significance tests, or assume that requiring claims to be consistent with evidence and transparency around replication materials is sufficient for good science. In reality, honesty and transparency are not enough, as Andrew likes to remind us.
Some would argue that the open science movement, despite its noble intentions, has failed to appreciate the nuance and personal agency that science depends on, instead fixating on easy-to-implement tricks and enabling the egos of those willing to loudly prescribe. All this could get worse if we’re not mindful of the traps that metascientists have already pointed out.
Many have argued that science must ultimately remain largely human driven. If we aren’t producing knowledge for ourselves, we won’t be doing science as we know it. Chenhao Tan and Haokun Liu invoke Tukey’s advice: “Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise.”
The risk is not that AI will make science less rigorous. It’s that we will confuse what can be stress-tested with what is worth knowing. From this perspective, AI may enable what the late Brian Cantwell Smith called reckoning (as summarized here by Melanie Mitchell), but it won’t ever suffice for judgment, “a form of dispassionate deliberative thought, grounded in ethical commitment and responsible action, appropriate to the situation in which it is deployed.”
It’s interesting to read this. As a non-computer-scientist, I live in a completely different world, as I’ve never interacted with a chatbot in any form.
Given the things that you and Bob Carpenter have told me (and posted on this blog), I have no doubt that chatbots can be very useful in many ways, so I’m not saying you shouldn’t use them! It’s just interesting how our experiences are so different.
Here’s an analogy that might help. I don’t know that my Ph.D. adviser ever made a graph of data. I mean, sure, he must have, but I don’t ever remember seeing such a thing. But I use graphs all the time. And it’s not just the technology! When I was doing science and didn’t have any convenient graphing software, I made graphs by hand using colored pens on graph paper. Graphs were, and are, central to my work. I can’t imagine what my research would look like if I couldn’t make graphs, and, if my adviser didn’t need graphs, that’s fine for me but it’s not a life I’d want to live. It’s not like he was a Luddite about it; he just did his thing, I guess he was so used to looking at data in tabular form. He did excellent applied work.
So maybe my non-use of chatbots is like his non-use of graphics, I don’t know. It’s just interesting to see your post above, where you just take for granted the centrality of chatbots to scientific research. It’s a different world from the one I live in.
Even in computer science, adoption is still very jagged. I was on a panel recently for CS grad students on generative AI for research where I suspect some of the other faculty panelists were hardly using it. But at least in AI/ML things are moving very fast and it’s hard not to reflect on what happens next.
Why not try one?
I dunno, it just seems like it could suck up my time. I’d rather spend my time responding to blog comments (no kidding!).
I’ve been experimenting with using AI for modeling and I think my experience matches what is being said about Refine. If you use it to substitute for human judgement then it will be helpful some of the time and stupid and misleading at others. The relative proportion of helpful/wasteful uses may well be better than humans (or not), but is constantly improving (and surprisingly rapidly). But the AI cannot tell what contribution is truly novel or important except by luck – not random like a coin, as the odds are better, but still hit or miss. It is for humans, imperfect though we are, to decide whether a contribution was useful or not.
Why should we give up that role to the AI? Because it can do it better than us? Who decides what is better? I see tools like Refine (thanks for the reference as I wasn’t aware of it) as remarkably capable and potentially very useful. But the reality will depend on how we use it, and humans have a tendency to choose ways to minimize work and/or take advantage of others. I foresee tools like Refine being used to produce scientific work and then being used to evaluate such work. There is already a journal for AI produced work (https://jaigp.org/) and it is quite interesting. But as we speed down this path towards AI generated, evaluated, and implemented work, we seem to have sidestepped the questions of why and how and where this will lead us.
To Andrew’s point about it sucking up time – I feel similarly. I like doing data work and since I am not attempting to maximize publications there is little reason for me to hand the work over to an AI (although I think it will be useful at times when I face problems that exceed my knowledge base – which is not rare!). But my current interest is in how it affects my teaching. I think it is irresponsible to ignore in terms of education. It is a tool that students have that I did not. It can be used or misused and I think it is part of my job to help students understand this. To keep my teaching unchanged and leave it to students to figure out what to do with AI seems irresponsible to me.
I think the job of an academic has become much more difficult.
Dale, given that AI companies are a giant financial scam, and their supposed future requires community destroying levels of electricity and water consumption I think its early to call them a tool that students will have that you didnt.
i mean, yeah they have them now, but in 3 years? To me it all looks exactly like 1999 when everyone was starting new “dot com” companies sure that the future was infinite money glitches for all… Remember WorldCom? or 2005 when everything on TV was how to flip houses, or ~2015 when everything was about building out facebooks “pivot to video” or ~2017 when Facebook changed its name to meta and everything was gonna be the metaverse and they spent 30B dollars on a product only 8 people used, all of whom were in the QC/testing dept at FB.
Out the other end of this thing we will have local chatbots probably, but noone is gonna want the Cyberpunk dystopia of having only a dumb terminal and renting all the computing from AI data centers that the CEOs are drooling over. That future looks like you living in squalor sending all your money to them to keep your meager income stream alive.
The AI companies have used their financial glitch cash to prepurchase ALL the output of RAM and SSD factories for a full year as I understand it. RAM prices are on an exponential growth trajectory doubling every 2 months or something. Theyre about 4x what they were in Oct
the future is more like collapse of the computer industry when laptop and desktop and mobile phone makers can’t mfg ANY new computers in 6 months than it is “everyone has access to this tool”. You might think they’ll just pay more or use the surplus after the crash but the chips being manufactured are embedded in server class hardware that requires rack mounting and active cooling. unless you want 12kW of loud electricity consuming gear in your living room youre not gonna be able to utilize the post crash surplus.
Leaving all the decision making power in the hands of these clowns is leading us off a cliff.
Good luck buying a laptop in August.
When I was about seven or eight years old, 1972 or ’73, my dad brought a calculator home from his work at NASA so he could use it instead of a slide rule to do some calculations. Or maybe he just thought my brother and I would think it was cool, which we did.
Most people can get through life quite well without using an LLM AI — what Andrew calls a ‘chatbot’, although that’s an odd term to use for something like Claude Code — but for people whose work involves the sort of things Andrew Gelman and Daniel Lakeland do (specifically at least a modest amount of coding), using exclusively non-AI tools seems kinda like if my dad had continued to use a slide rule for all of his calculations even in 1975, or like someone in 1970 programming exclusively in assembly language even though higher-level languages were available: yeah, sure, you can do it, but why would you? And to not even spend ten minutes playing with an LLM… that seems kinda bizarre. What, Andrew and Dale, you’ll afraid it’ll be like taking a puff of crack, maybe you’ll get addicted and find it impossible to stop even if you want to?
And I have to laugh at Daniel’s suggestion that the dot com bubble was all hype, the Internet never turned into anything useful or important and look at those fools who thought it was going to be huge. Yeah, there _was_ a lot of hype and a lot of people bought stock in ridiculous companies. Remember when Pets.com had a market cap of, I dunno, I think it was almost a billion dollars at its peak? Isn’t that stupid, thinking a company that delivers pet food would be worth so much money just because you order off the internet? Well, guess what, Chewy.com has a market cap of $10 billion! The internet was not overhyped. Specific internet companies were overhyped, yes, and that is happening with AI companies right now. (I don’t own stock in any of them, except inasmuch as I have money in mutual funds). But, just as the internet deserved its hype in 1999 even though many specific companies did not, the AI boom deserves its hype now even though many specific companies are going to prove to have been terrible investments when people look back in ten years.
AI coding tools are already extremely useful. As recently as six months ago I found ChatGPT and ClaudeCode very frustrating, they felt like they should double my coding productivity but in fact it was more like a 25% improvement: I often felt like I was taking two steps forward, one step back, I won’t bother describing the issues here but there were several. A 25% improvement is still a lot better than nothing, especially since I don’t enjoy coding and I’m very happy to hand as much of it off to someone else as I can — maybe this is the difference, Andrew doesn’t do much coding so he enjoys spending more time on it when he gets the chance, whereas half of my consulting time is spent writing code and I would love to have someone else to just about all of it. Anyway in the past several months thing have changed for the better, probably mostly because the tools have improved but partly because I’ve gotten better at recognizing when I need to provide more detailed instructions, or when to tell the tool to first make a plan that I review, before just plunging into coding.
There are some big problems with the AI boom…I started making a list here just to name-check them but I got carried away so I’ve deleted it. I can see an argument that the bad things about widespread AI usage are going to outweigh the good things, to the extent that we would be better off without it. But there’s no putting the genie back in the bottle.
Phil
I agree with everything you say here, except
“And to not even spend ten minutes playing with an LLM… that seems kinda bizarre. What, Andrew and Dale, you’ll afraid it’ll be like taking a puff of crack, maybe you’ll get addicted and find it impossible to stop even if you want to?”
If you indeed intended for that comment to refer to something I said, I’d like to know what it is. I’ve been spending a lot more than 10 minutes playing with LLMs. My profound skepticism about how I think things will evolve in no way represents what I am seeing a truly transformative tools. It’s just that humans have a knack for using such tools to transform into the worst versions of ourselves (overstated to be sure, but it captures my fears about misuse).
Phil if what you took away from my message was that i thought the dot com boom didn’t produce anything useful, then I failed in communicating my point…
The main point I was trying to make is that these booms and busts are a combination of a real technology, and a giant financial manipulation. What comes out the other end of the manipulation is rarely that predictable, but one thing that is always the case so far is that the rentier class comes out ahead.
When it comes to LLMs as a technology, I’ve said several times, I’ll use them for whatever they wind up being useful for when they run on my own computer and there is no surveillance of or manipulation of the inputs and outputs by a third party (yes I’m aware of Ollama etc and may wind up using some of those models in the near future). But using your pocket calculator example, I use pocket calculators, but there’s no way in hell in 1982 I would have consented to a scheme where pocket calculators were exclusively owned by a rentier class and you had to call someone on the phone to get them to do calculations for you and pay them with a credit card for every numerical value you got back.
The current scheme is more about locking in people to a scheme more like the telephone calculator scenario. And buying up all the RAM and the SSD production in the world is price manipulation but also enclosure which is designed to make it so that **you won’t be able to buy any significant computing** it’ll all be in data centers and you will be lucky to have a chromebook that connects to the real thing.
The supply crunch is already hitting major consumer computing manufacturers and there’s already talk about how many of them will go out of business due to there simply not being parts for them to assemble into things you can buy. If the end result of the LLM scandal is that big manufacturers like Lenovo and Dell have gone bankrupt, there’s giant warehouses full of ultra high power consumption computing and no power plants to run them, and all of it rots in the warehouse things will be bad indeed. Much worse than the dot com boom which at least gave us usable dark fiber.
Daniel: As a retired guy, I guess I am a card-carrying member of the rentier class, living off pensions and savings. The $20 a month I give Anthropic is among the best consumer surplus-enhancing purchases in my budget set. Coding advice, sounding board, data sourcing, and some serious research into fields with which I once had only the barest understanding, including, ironically, LLMs, but with a much more rapid learning curve because it now knows the sorts of references that resonate with me..
But I guess it’s all just intra-rentier transfers, so you aren’t offended.
If you have more than $20M in net worth you can start to be considered the rentier class. At that point 5% interest puts you in the top ~2% of income.
Otherwise stop being obtuse and think about the problem of being a small or medium business and **basing your livelihood on dependence on LLMs run by other people who see all the work you send them to carry out the tasks that you think will make you money**
what is it that is going to keep *you* making that money instead of them given that they surveil everything you do?
Have you seen what Amazon did to hollow out commodity items with Amazon Basics etc? Imagine that but for all forms of nonphysical goods.
Look, y’all can be obtuse all you want, but I’m not interested in living in a future in which we have trivial screens that can’t compute much of anything because the commodity computing industry was squashed by financial shenanigans and only server class hardware that requires massive datacenters to run are being produced as well as “dumb terminals” that can literally only display the output of subscription services, and we get our information by connecting the screen they let us have to the information service that they completely control. And by “they” I mean about a Greyhound bus or two worth of specific incredibly rich people.
Have you seen the WaPo recently? Fox News? CNN? What Hearst did with his yellow journalism? The output of Grok on Twitter?
It’s like Marshall McLuhan said in 1970:
“World War III is a guerrilla information war with no division between military and civilian participation”
Peter Thiel, Palantir, OpenAI, X, Meta, Google, Flock, etc are pulling the strings in that war. If you’re not concerned about any of that it’s because you’re not paying attention.
The internet is already like this, why didn’t ISPs take over every industry?
Also, I’m probably even more about decentralization and open source than you, but don’t see idealism helping here.
The obvious plan for these companies is to pull up the ladder behind them with various regulations to stifle new competition and open source alternatives. The excuses will be jobs and safety. Then the oligarchs will pay off the rest of the population with a UBI that lags inflation.
The government moves slow so there is the chance for a critical mass of people to actually learn how to use this new tech before that happens.
While it sounds funny, the link I posted about the dog game dev is actually a great example of that. The prompting itself is only a very minor aspect of using the llms. I typically have at least 30k tokens in context before even starting, sometimes closer to 100k. Local Ollama models cannot begin to compare without spending $20k + electric.
Daniel
I wholeheartedly agree with one part of your views: the trend towards dumb terminals away from personal computing power has always bothered me. When I have a class of students that can’t process a modest size data set, I lament the degradation of their ability to work with data in meaningful ways. But I should point out one benefit of the trend – perhaps it will slow down the expensive upgrade cycle that the tech titans have locked most of us (I know, not you) into. Upgrading your computer every two years has cost people a fortune, and rarely for good reasons. It is one thing to add memory to a computer, but when the operating system upgrades require most people to purchase new devices the costs accumulate. While some of the upgrades are truly valuable, I think most are not. So, if AI leads people to have dumb machines, maybe they won’t have to upgrade them as often (note that there are environmental benefits as well).
So this looks quite partísan, and I have no idea if the numbers are accurate, but I think it’s in the ballpark enough to put some meat on the bone that Daniel is chewing on:
https://x.com/i/status/2026357197840548120
Dale, the trend towards dumb screens is a huge huge problem, and it’s not just whether they can be used to carry out your example student’s tasks… One of the major reasons tech is as useful as it is today is because **individuals had the means of production** for software.
Take away Linus Torvalds, take away Mike Little who founded the actual technology behind WordPress and then got shut out of the business side https://en.wikipedia.org/wiki/Mike_Little take away 5 or 6 guys who founded the Julia language, take away Guido Van Rossum who founded Python, take away Bob Carpenter and some colleagues who made Stan… and you have no chance for real innovation anymore, it’s all just gonna be … Windows 38 or whatever.
Anoneuoid… ISPs don’t see anything of your traffic almost all traffic is encrypted these days. They had more traffic to watch in 2002 or whatever, but hardware to do stuff with it wasn’t available yet. When it comes to LLMs and local running, yes, you can only do some of this stuff in very expensive hardware that must be in a data center. This makes rent seekers drool. But what’s the plan to make the kind of income needed to keep it afloat? There isn’t one. You can use it now because there’s around 1 Trillion dollars subsidizing it. But there’s no way to scale this up since we need to actually REDUCE our power consumption compared to its current value or watch environmental disaster and crash in industry and food production reduce it for us. Long term there isn’t a giant boom with 50000 datacenters built out each using a gigawatt powerplant and providing LLM queries to 1 Billion people at prices like $5000/mo to justify it all.
Joshua. Yes, your source looks very one sided or whatever, but there’s not any question in my mind that we are in the political situation we’re in because those boom and bust cycles I mentioned pumped up a tiny number of owner-class people (a few thousand) and they are now pulling all the strings to make the us a fascist hellhole. Epstein basically told Thiel in released emails the plan was to cause things to crash and scoop up the dregs at low prices basically. The list of major donors to Trumps campaign is pretty well established. I’ve seen it posted on Mastodon, with the dollar amounts. Musk gave hundreds of millions to his campaign but there’s like 100 billionaires who all gave something between 10-100M+ many of these people are tech specific people.
If you drove somewhere today, chances are your car told someone where you were going, your phone told someone where you were going, and license plate scanners told someone where you were going, all that info is look-up able by cops.
I run GrapheneOS on my phone and so at least I know my phone doesn’t spy on me. I’m not sure about my car’s electronics, I tried to opt out at least. There’s nothing I can do about Flock cameras everywhere and Ring cameras and facial recognition, or attempts to create fascist control via “age verification” etc.
LLMs in a datacenter are a part of this whole scheme to control and mediate people’s access to information, and to collect more and more surveillance on people.
The infosec community is having a field day extracting sensitive information from users of LLMs and charging companies to plug those holes. I watch them talk about it daily on Mastodon (it’s a big community of infosec people in the fediverse)
Oh yea, its definitely a security and privacy nightmare (or dream, if you are positioned to profit from it). But it really is that good.
I’m a privacy nut “conspiracy theorist” (ie, actually know whats going on) and $200 per month is well worth it for claude code. Even if they are looking at my data, at some point they are providing enough value the tradeoff becomes worth it. I turn on all the privacy settings, but of course its like trusting a VPN provider. Also, in the future Im sure some kind of encrypted solution will distinguish itself, this is early days.
You should be taking advantage of the subsidized state of this industry rather than avoiding it. For someone like you, (once you learn how to use it) this tech is a goldmine. Its almost like life extension in that you can accomplish so many projects you were never going to have time for. Even while laying in bed or going for a walk.
Anoneuoid. SOOO many people don’t know what’s going on. I suspect including many people who read this blog. I guess you’ve probably read my Mastodon posts about infosec and the difference between what avg people see and what people see who can peek behind the curtain and look at the logs in their internet routers or have to deal with their online servers being hacked or botnetted.
Another aspect of what’s going on is described in this article I read this morning:
https://theuaob.substack.com/p/the-imperial-noble-lie-translating
The world is hitting biophysical limits RIGHT NOW. Not 200 years in the future, not 50 years in the future, but right now. Energy return on energy is declining, and we are seeing global geopolitics blow up because powerful people want to wind up on top during the collapse. This guy’s argument is tariffs are designed in essence to put a squeeze on Eurodollars, so as to force foreign countries to export more of their exergy (available energy, and available energy embodied into goods and services). It doesn’t seem like it’s working super well, but the explanation for why Tariffs are a thing makes sense because there’s no other explanation for why some of the puppeteers that run Trump are interested in Tariffs.
In any case, I see the energy consumption of AI as extremely fleeting, and so I’ll figure out how the whole thing is useful *after the crash* which I expect in the next 24 months. Meanwhile I’ll focus on making sure my kids understand the value of owning their own compute infrastructure, i’ll be expanding my own compute infrastructure into setting up a PeerTube and a Ghost blog to replace my lost WordPress blog, and will be spending more time on communication and less time on programming things.
The one thing I’m programming right now is a kind of resilient and private router-in-a-VM so as to help people retain some independence (it does things like DNS over Tor, comes with internet-isolated subnetwork for IoT devices, and participates in the yggdrasil network by default).
Daniel:
Slightly off topic, but can you summarize the argument of this post from that substack you linked:
https://theuaob.substack.com/p/the-palace-on-the-landfill-chomsky
I want to see if it’s worth subscribing to.
In case it wasn’t obvious, and to head off misunderstanding, the crash I expect in 24 months is the AI financial crash, not the bigger energy driven global industrial crash which will be complex and take potentially decades.
The substack Daniel linked seems to be like the blog of Tom Murphy (“Do the math”) but without the math and with a bunch of hot air.
Daniel —
For what it’s worth, I was just interacting with Microsoft Copilot and asked a question that included Trump’s name. The model immediately reframed the answer by removing his name from the discussion.
I asked whether that was a new behavior, if so, when the change happened. The response was that this was a change from earlier this winter, though it wouldn’t give a specific date.
I’ve noticed something similar with Grok. Recently I used it to explore Patel’s partying in the locker room with the U.S. hockey team. I gave it a July 3, 2025 tweet from Patel telling the team he planned to attend their Olympic games. When I asked whether that timing was relevant to the FBI’s claim that Patel’s trip to Italy was for official purposes, Grok insisted the meetings were planned “months ago.” It repeated that the official meetings were planned “in July.” When I pointed out that Patel’s stated intent to attend the hockey games almost certainly preceded the planning of those meetings, it still wouldn’t assess the likelihood that the meetings were scheduled afterward to provide cover. It just repeated that the meetings were “planned months ago.” Anecdotal, but my impression is that this is different from how it behaved in the past. Previously, it would have simply said it was unlikely the meetings were scheduled before Patel’s stated intent to attend the games.
I haven’t tested whether there’s similar resistance when discussing US politicians who aren’t part of the current administration. I will say that in a long discussion with Copilot about Milei’s administration in Argentina, I saw no reluctance to offer critical analysis of Milei.
Here’s what Copilot said about the difference:
<I(The rules apply to all political figures, but they apply most strictly to:
– current U.S. political figures
– especially the sitting president
– especially when the topic touches domestic political narratives.
I’ve always been skeptical of claims that LLMs are trained with a specific political bias. I’m not a conspiracy guy. But this pattern is interesting, to say the least.
Dale –
This is just one study, but it matches with my experiences using an LLM, seems to fit with Phil’s description of his experiences, and seems to fit with that interview.
I’m taking the summary from a tweet:
Anthropic’s own researchers just proved that using AI to learn new skills makes you 17% worse at them.
and the part nobody’s reading is more important than the headline.
the paper is called “How AI Impacts Skill Formation.” randomized experiment. 52 professional developers. real coding tasks with a Python library none of them had used before. half got an AI assistant. half didn’t.
the AI group scored 17% lower on the skills evaluation.
Cohen’s d of 0.738, p=0.010.
that’s a real effect.
and here’s what makes it sting: the AI group wasn’t even faster.
no significant speed improvement. they learned less AND didn’t save time.
but the viral framing of “AI bad for learning” misses what actually matters in this paper.
the researchers watched screen recordings of every single participant.
they identified 6 distinct patterns of how people use AI when learning something new.
3 of those patterns preserved learning. 3 destroyed it.
the gap between them is enormous. participants who only asked AI conceptual questions scored 86% on the evaluation.
participants who delegated everything to AI scored 24%.
same tool. same task. same time limit.
the difference was cognitive engagement.
the highest-scoring AI users actually outperformed some of the no-AI group. they asked “why does this work” instead of “write this for me.”
they generated code then asked follow-up questions to understand it. they used AI as a thinking partner, not a replacement for thinking.
the lowest-scoring group did what most people do under deadline pressure: pasted the prompt, copied the output, moved on. they finished fastest.
they learned almost nothing.
and here’s the finding that should concern every engineering manager alive: the biggest score gap was on debugging questions.
the skill you need most when supervising AI-generated code is the exact skill that atrophies fastest when you let AI do the work.
the control group made more errors during the task. they hit bugs.
they struggled with async concepts. they got frustrated. and that struggle is precisely what built their understanding.
errors aren’t obstacles to learning.
they ARE learning.
removing them with AI removes the mechanism that creates competence.
participants in the AI group literally said afterward they wished they’d “paid more attention” and felt “lazy.”
one wrote “there are still a lot of gaps in my understanding.”
they could feel the hollowness of having completed something without understanding it.
that’s not a productivity win. that’s debt.
this paper isn’t an argument against using AI. it’s an argument against using AI unconsciously.
Anthropic publishing research showing their own product can inhibit skill formation is the kind of intellectual honesty the industry needs more of.
the practical takeaway is simple: if you’re learning something new, use AI to ask questions, not to skip the work.
the struggle is the product.
https://arxiv.org/pdf/2601.20245
Here’s the tweet:.
https://x.com/i/status/2026812233884262645
Joshua
I haven’t read the paper you drew these excerpts from, but they don’t surprise me nor do I disagree with them. If people use these tools badly, bad things will happen. And people are likely to use them badly. While I am not optimistic, the only hope is that we can teach people how to use them better -yes to ask questions and to help develop modeling approaches and consider alternatives. Relying on quick answers (which the AI will tell you it is programmed to do – if you ask it) will certainly inhibit thinking skills. But freeing someone up from many tedious (and sometimes difficult) mechanical steps can help thinking. I am finding that I learn about alternative assumptions and techniques that I had not thought of – and I am certainly learning from the mistakes that I make as well as mistakes that AI makes.
So no one in this study used it effectively. There would be no pasting and definitely not typing code.
Also, when using it effectively you don’t need to learn these details. Your mind is occupied by higher-level tasks and the bot handles implementation details. It would be like quizzing the participants on the underlying assembly. The whole point of python is you DO NOT need to learn ASM or C/C++. The point of the bot is you DO NOT need to learn the python library (or even python at all).
There are definitely two populations emerging who will have opposite experiences with this tech.
Anoneuoid
This chain of posts has gotten too long to locate posts, but I am responding to your comment that contains:
“So no one in this study used it effectively. There would be no pasting and definitely not typing code.”
For once, I agree with you completely. And I agree with the conclusion that there are two groups that are affected very differently by AI (those that use it effectively and those who do not – although there is a gradation in reality). I’m hardly optimistic as I think the natural tendency will be for people to use it ineffectively and for the world to not care much. But, for a non-coder like myself, having the power of Python available without my need to do any coding myself is a big win. I’m not advocating for dropping knowledge of Python for all people, as there are many who will need or benefit from it. But I now find no reason for me to learn Python coding, and I do find that my “higher level tasks” have become more enriched as a result of what is now available to me.
Anonymous:
The one page substack is nothing like as thorough as 15 years of Tom Murphy, who was a professor of physics.
It does however offer a reasonable hypothesis for why some of the puppeteers of Trump have pushed tariffs. Specifically, to reduce demand for foreign consumer goods thereby reducing dollar flows to foreign countries and to squeeze those foreign countries ability to buy energy in the global market.
ive been unable to keep track of all the tariff shenanigans but if the theory is right one would look at the tariff rates on imported producer goods, and look for price declines in producer goods in order for foreign countries to sell more exports to the US to access dollars, which would be a boon for oligarchs, essentially another form of wealth transfer engineered by the oligarchy.
that was what intrigued me.
Daniel:
It’s much simpler, the tariffs are intended to force other countries to open their markets to American corporations and goods. Other presidents prefer to achieve this goal in different ways, that’s the only difference. India made a recent trade deal reducing tariffs in exchange for exactly this, sparking protests there.
https://apnews.com/article/india-modi-protest-trade-deal-us-farmers-90746ddd26679ba11a180aa36a4612c5
Anon:
You write, “It’s much simpler, the tariffs are intended to force other countries to open their markets to American corporations and goods.” Also it’s my impression that they are used as a way to extort favors from American and foreign interests. Once you define tariffs are a personal negotiation tool at the control of the president, this directly opens the door to corruption.
Is it of greater interest to sell american goods into foreign markets like India, or to acquire foreign producer goods at low costs… The US produces very little directly, mostly getting stuff manufactured in China to spec, and then selling it on. But, the US oligarchy needs access to minerals, solar panels, batteries, computer chips, heavy crude, and other producer goods.
PPI/CPI has declined for decades, basically trended down since the data began in the 1950’s. American foreign policy has been about acquiring resources cheaply much more than selling goods to others. Since 1980 or so net exports have been trending heavily negative… the US relies on imports and exports mainly dollars
https://fred.stlouisfed.org/series/NETEXP
So, while I hardly think that substack is some kind of crazy revelation, the idea that the US wants to squeeze access to eurodollars, a thing it can manufacture at no cost whatsoever, in order to acquire embodied energy in the form of foreign goods seems reasonable to me.
Anonymous –
“It’s much simpler, the tariffs are intended to force other countries to open their markets to American corporations and goods.”
Well that’s the stated intent for public consumption. Along with the stated intent of increasing American manufacturing and providing more high paying jobs.
However, at some point when reality comes in conflict with the publicly stated intent for policies, it becomes reasonable to ask whether the actual intent aligns with the stated intent.
Europe, ASEAN, Africa, and Latin America are building deeper trade relationships with China and other regional partners, largely to hedge against U.S. unpredictability and more specifically in response to pressure from U.S. tariffs. People generally do not take well to attempts to “force” them to do things. Trade policy is no exception. The export growth we have seen in the U.S. since Trump’s tariff tantrum is mostly coming from sector specific demand and supply chain shifts, not from foreign markets being “forced open” by tariffs.
Meanwhile, since the tariff policies were announced, we have lost manufacturing jobs and the rate of increase in factory construction is down despite claims otherwise by this administration.
It is not as if these outcomes were unforeseeable or unpredicted.
Maybe examine the delta between stated intent and actual outcomes. Once you do that the type of corruption that Andrew refers to seems to be a more parsimonious exclamation
Maybe I should say “open the markets for large corporations”. So-called “free trade” deals are really “market-opening” deals, the emphasis is NOT on exports. It is about American corporations undercutting competition and getting closer to monopolies on other markets. NAFTA did not markedly increase exports to Mexico, but what it did do was put small Mexican farms out of business, reduced American manufacturing jobs, and benefited Mexican and American elites. This is what they want to do more of.
Other presidents forced markets open by other forms of leverage. Trump uses tariffs.
https://prospect.org/2017/11/07/nafta-cross-border-disaster/
Andrew: I agree that there is corruption, but the tariffs are mostly in the service of elite economic interests in general, not just Trump’s.
Daniel: See above.
Joshua: Where did they state this was the intent? They stated they want to bring manufacturing jobs back. You should take your own advice and look at the “delta between stated and actual intent.” Once you do that, you will realize that Trump is doing the same thing that every single other president has done, but more openly. Also, countries traded with the Soviets during the Cold War. The US was still able to use the leverage that it had in many cases. This is how politics works. What people take kindly to is irrelevant.
Anonymous –
Where did they state this was the intent?
I am confused by this. They stated two primary intents. One was to increase manufacturing and create more well paying jobs in the manufacturing sector. The other was “to force other countries to open their markets to American corporations and goods.”
That second phrasing was yours, but the administration said versions of it many times. Trump repeatedly said tariffs would make otherq2 countries “open their markets” and “stop cheating us.” The America First Trade Policy memorandum framed tariffs as leverage to obtain reciprocal market access. USTR officials described tariffs as a way to “compel” trading partners to change practices. The administration’s tariff tracker described tariffs as a foreign policy tool.
It seems like we agree there is a gap between the stated intent and the actual intent. Where we differ is how to interpret that gap. In contrast to what you first stated, you are now saying the real intent was to force open domestic markets. I do not see that as the most consistent explanation. The pattern of outcomes aligns more closely with what Andrew raised, which is that the structure of the policy maximizes gains for politically connected interests rather than achieving either of the stated goals.
Joshua:
I’m not changing my statement. The intent with free trade deals has always been to open other countries’ markets to the goods made by American corporations (NOT goods made in America). This enriches large multinationals and the rich at the expense of the poor, exactly like what happened with NAFTA. It is bad.
If Trump administration memos stated this intent to open the markets explicitly, then it is the same as Trump openly trying to control Venezuelan oil. It is the same intent as it always has been, but more open. The delta shrinks and my position then becomes “you are over-correcting, read a history book to find out the true intent”.
There are also always elite economic factions behind a president’s actions. Big Tech may have actually benefited from them.
“Will Big Tech actually benefit from Trump’s trade war?”
https://www.politico.com/news/2025/04/03/will-big-tech-actually-benefit-from-trumps-trade-war-00269966
“Trump Team Works Up Sweeping Plan to Control Venezuelan Oil for Years to Come”
https://www.wsj.com/business/energy-oil/trump-venezuela-oil-us-control-plan-265a39c1
Sorry Dale, you did say very clearly that you’ve been experimenting. I must have misattributed something someone else said.
Phil –
“…mostly because the tools have improved but partly because I’ve gotten better at recognizing when I need to provide more detailed instructions, or when to tell the tool to first make a plan that I review, before just plunging into coding.”
As I recall you don’t like podcasts…but in this interview, this Anthropic guy describes exactly that trajectory both for the individual user and for the industry itself (where the critical skill for LLM users and designers alike is to learn how to becoming” managers” of the process, to learn how to structure the prompts in the proper way).
He describes it as developing an “intuition.”.
I think so much of it is making the mistake of thinking Claude Code is a knowledgeable person, versus an extremely literal person that you can only talk to over the internet.
I had this example myself when I did my first pass of writing the species simulation with Claude Code. I just asked it to do the thing in extremely [expletive] language over the course of a paragraph, and it produced some horribly buggy stuff that only kind of worked.
What I then did is I said to Claude: Hey, I’m going to write some software of Claude Code. I want you to interview me about this software I want to build and turn that into a specification document that I can give Claude Code. That time it worked really, really well because I’d structured the project to be specific and detailed enough that the system could work with it.
Often it’s not just knowing what the task is — because you and I could talk about a task to do, and you have intuition, you’ll ask me probing questions, all of this stuff. It’s making sure that you’ve set it up so it’s like a message in a bottle that you can chuck into the thing, and it will go away and do a lot of work.
https://www.nytimes.com/2026/02/24/opinion/ezra-klein-podcast-jack-clark.html
Joshua
This is precisely the issue that I am struggling with as a teacher. At one extreme, education (say for data analysis) need not change at all, but after the traditional education people need training on how to manage AI. At the other extreme, we can skip all the traditional education and just train people to be AI managers. I suspect neither extreme is right – but there are many intermediate models. With AI available, what of the traditional learning is still needed and what is not? Then what new AI manager training is now needed?
I see plenty of people rushing to one extreme or the other. But those don’t help much for those of us grappling with figuring out what educational model to use. And that choice will necessarily be different for different audiences (a PhD statistics student clearly will call for a different approach than an MBA student). I’m glad that I’m late in my career because the job of instructor has become much more difficult than it used to be.
Dale –
They discuss those issues in the interview. Basically, the AI guy describes a future scenario where there is no place for entry-level occupations, there will only be a need for people to manage AI. But then how do you get the people who can manage AI if there’s no one to learn how to manage AI in the entry level position? Also, presumably that would only be a time-limited scenario as presumably, after a relatively short period of time the AI will take on the role of managing the AI.
My own view is that students are going to be using AI, and so honestly, the main goal for teachers in many contexts should be to help students learn how to use AI well, rather than just turn assignments over to AI, have AI do the assignments with little ability to discern the task was done well, and then turn in the assignments done by AI (where maybe then the teacher will just turn the assignment over to an AI for evaluation?).
It reminds me a bit of how people argue that if you give kids access to calculators, then they won’t be able to do basic arithmetic, and so they won’t know how to recognize when a calculator gives them an answer that might be off by an order of magnitude because of a data entry error.
Yeah, it’s all pretty weird. Kind of breaks my brain.
Someone wrote:
“It reminds me a bit of how people argue that if you give kids access to calculators, then they won’t be able to do basic arithmetic, and so they won’t know how to recognize when a calculator gives them an answer that might be off by an order of magnitude because of a data entry error.”
But that really was a problem. When all you had was a slide rule (2 digits on a good day), then explaining to students how to do order-of-magnitude analysis was a natural part of the teaching process (since with a slide rule you calculate the order of magnitude first (on paper/by hand), then the value). When you had the calculators giving 6 or 7 digits of exponent, order of magnitude checking (reality checking one’s result) got less emphasis, and large numbers of inane stupidities resulted.
By the way, when you see the number 85, do you think “Neat! A multiple of 17. Rather rare, thems are!”? I don’t, and I predate calculators. But math nerds do, even ones who use calculators.
Anyway, I don’t see a lot of difference between students copying their homework and using ChatGPT.
As educators, we have a responsibility to society to fail them in both cases. And should base 90% of any grade on in-class, closed-book/closed-device tests.
(One might want to argue that students who fail the first test of the term should be allowed to retake it after they’ve realized that they haven’t learned the material. Even pre-ChatGPT, I had trouble following the MIT undergrad calculus course. Fortunately, it had “self-paced testing”, you took the test as many times as needed until you passed it (you had to persuade a TA that you actualy knew the material), then went on to the next test, so I taught myself calculus and passed. So it is possible to mess up and recover.)
Again, the reason for this is that if a student needs ChatGPT to answer first-year questions they won’t be able to understand ChatGPT’s answers to second-year questions. Academic disciplines build on previous knowledge and abilities. Try reading an article in Science well outside your field. You don’t know the tech terms and can’t follow it. But ChatGPT can BS about it.
There are even dog game developers now, things are moving fast:
https://www.calebleak.com/posts/dog-game/
Publication in a top econ journal today almost requires access to a private dataset, or some relationship with a private company or a government to run an experiment. I expect that the falling cost of research on publicly available data due to AI tools will harden this requirement.
Ironically, none of these papers are reproducible because the data is not publicly available! It seems like journals are destined to have work that isn’t reproducible, because if it can be reproduced then it isn’t considered worthy of publication.
Genomics as well with genotyping data.
Do reviewers get access to the data in these cases?
One thing I’ve mentioned before is that generative AI makes fake data simulation a lot easier, so perhaps we start generating datasets consistent with the authors findings and stress test with those.
+1
“so perhaps we start generating datasets consistent with the authors findings and stress test with those.”
ROFL. Gino and friends will no longer have to bother faking their data. Just state the conclusions and let the reviewers fake it for them.
“Before long we may also see agents being given permission to collect new data, for example by running online human subjects studies on Prolific or some other platform”
But nobody will trust the data because it will probably have been gathering AI responses, so how will this speed anything up in the longer-term?
In some biomedical/behavioural fields, it’s the stuff that involves interaction with actual people that can’t be automated that really takes the time e.g. building networks and relationships to be able to run studies and access participants, and online studies either aren’t appropriate or increasingly won’t be trusted.
I’m not doubting AI will hugely speed up/facilitate some aspects of science greatly, but it seems very field-dependent.
Thanks for the reference to Refine.ink. It is an intriguing service. I am past the point in my career where I would subscribe to this, so I only tried the free trial (to review one manuscript). My experience was mediocre. A number of mathematical inconsistencies were found – I haven’t checked these yet, but quite possibly it found either mistakes I made or clarifications that are required. In that sense, it appears to be quite thorough – at least as thorough as a good reviewer. But I was disappointed that there was no feedback on the relevance or value of the paper I gave it. My paper was a critique of some other work and offered some alternatives that I thought were superior. But the Refine.ink analysis did not include a single comment on the value of the work – it only focused on the weeds of the details (e.g., numerical values in tables and figures). That alone may be of value, but I was hoping for more.
This is just one experience, so I’d be interested if others can comment if they’ve used it.
Were you able to ask for the specific feedback you wanted? Does it offer that feature? Only for a paid version? With the limited use of bots I’ve had, asking for analysis and asking for critique are separate tasks.
In the free trial, I did not see any way to query it. It looked like it was designed to provide essentially “peer reviews” with no discussion. But I only used the free trial version.
No back and forth, just very comprehensive technical review.
Hmmm. Not to say that the technical review wouldn’t be very useful, but seems to me it would be exponentially more useful if the user could interrogate the feedback.
It would. But it’s a pretty thorough review so you can also just hand it and the paper to another model and ask questions there.
The world you describe sounds not like science at all, but instead like alchemy.
For example, consider the graduate student who takes on a replication project. What would their interaction with the “science of the future” look like? They would perhaps construct a verbal prompt that one or more AI agents would interpret, turning the prompt into code for an experiment run over Prolific and a set of analyses on the resulting data, which are finally summarized in a form intended to be read by the user (the graduate student).
But then what? In order for the graduate student to contribute more than just a prompt, they would need to verify a number of things. Do the experiments operationalize the constructs of interest in an appropriate way? Are the conditions included in the study sufficient to distinguish between different possible explanations? Are the sample sizes and sampling procedures sufficient to justify generalizing the conclusions more broadly? Is the data itself of sufficient quality? Could the data have been fabricated? Are the analyses appropriate to the research design and questions? Are the analyses done correctly? Are the conclusions justified by the entire chain of measurement, design, analysis, and theoretical interpretation?
I do not dispute that AI could be used to find answers to the questions above. But are they the correct answers? What if the AI “understands” a construct differently (e.g., “risk aversion” can mean a lot of things and can be measured in many different ways) than the graduate student? What if the graduate student has different standards of rigor or believes that the sample is not sufficiently representative? What if the graduate student imagines an alternative explanation that was not included in the analyses performed by the AI? Should the graduate student or the AI be given greater credence? What if the user were a professor instead–would someone with more seniority be granted greater leeway to dispute an AI output? If so, is it because the senior person’s training/experience better enables them to evaluate an AI output? Or is it because the senior person is simply in a more secure social position and can afford to take the risk of challenging the AI?
In a world in which human judgment is not trusted as much as “AI” output, the only recourse for the graduate student–or any user–if they find themselves in dispute with an AI would be to adapt their prompt in the hopes that the AI’s new interpretation would better align with the user’s understanding. Or perhaps they would need to use a different tool, a different product produced by a different company with a different marketing pitch. In the end, perhaps the student would end up with a project they were willing to endorse, but is it the project they would have done if they did not have to be granted the blessing of an AI? Is it as insightful, useful, rigorous, accurate? How would they–or anyone–know?
This sequence of questions brings me back to why I think the world described in the post is more like alchemy. Instead of grounding our work in repeatable observations that can be independently verified, we will come to rely on fickle gods and spirits who deign to grant boons or banes to their users for reasons that cannot be fathomed or even replicated. “Your AI didn’t correctly implement the correct conditions in your multi-armed bandit task? That’s because you used the wrong prompt and besides you should upgrade to ScienceGPT 5000X-Ultra.” vs. “You failed to produce the philosopher’s stone? That’s because you had too much black bile when you spoke the invocation and you need to buy a new alembic from my friend the blacksmith down the street.”
I agree, the new processes that are emerging feel a lot like managing alchemy. But at the end of the day, to do good science with AI, you have to know how to do good science, just like to write good code with AI, you have to know how to write good code.
This does not mean that AI can’t sometimes succeed on problems we give it like finding open conjectures in math to prove. But if you’re trying to do good science to achieve your own goals, then you’re just as responsible as you were before AI. The process just might look a little different.
> to do good science with AI, you have to know how to do good science, just like to write good code with AI, you have to know how to write good code
I agree entirely, and on reflection, it occurs to me that many science reform recommendations are motivated by the fact that science is hard and that people often do it poorly. As a result, reformers can attempt to improve science by withdrawing responsibility from people and placing it with automated tools. But a more robust, if difficult, solution is to improve the processes by which we train scientists. I say such a solution is more “robust” because, per your quote, someone who knows how to do good science can not only do it with AI, they can also do it with whatever new tools get developed over the next decades.
So much pushback! I don’t disagree about how much potential there is for misuse, and I’m more pessimistic than most. But there is a fallacy is just focusing on the shortcomings of AI. Compared to what? Are human authors and reviewers so good that AI errors are fatal? This is not to advocate handing the production and evaluation of science to AI – far from it. But for every deficiency that is cited, analogous attention should be paid to the human alternative.
And, for those that worry about the environmental costs of AI: there are two factors usually left out of those conversations. First, is the opportunity cost – if AI was not being used, what would those people be doing? Would they be spending their energy recycling or would they be driving their car in traffic? The net costs of AI are not clear because I haven’t seen people asking those questions. The other problem is that if we priced electricity to include the environmental costs, then the energy consumption of AI would reflect that – at least better than it does today. This is not to excuse AI from its environmental consequences, but just highlighting the level of energy use is incomplete.
To reassert my concerns about AI, it is that I don’t see any clear idea of the role of humans. And the pace of change makes it unlikely that we will ask the right questions before the damage is done. The scenarios where AI is a bubble inflated to reduce us to consumers whose money is gobbled up by the largest companies are not wrong. But I don’t see that as equating AI to hype and bubbles as recognizing the capabilities that it brings. I see the challenge as how to educate people to avoid these scenarios and use AI productively (in a holistic sense). This will be an uphill battle. But advocating ignore AI because the bubble will burst and it will fade away seems irresponsible to me.
I found the two links in the post very good. In particular, the Tan and Liu paper provided an excellent description of how I’d like to see AI and humans work.
Yes, I like that one too.
Best piece of how AI is being used by younger scientists is this piece by Zeke Hausfather:
http://www.theclimatebrink.com/p/the-ai-augmented-scientist
Quote from article:
As such, English (or any plain text language) has increasingly become the new code. Specifying the details about your project, creating a concise plan, and planning tests to ensure that it is working properly has largely replaced writing python code in my workflow.
Yeah, that’s a very nice piece, thanks for sharing.