AI Disarray

Today is Stanford’s commencement. It has always been a special day every year. I wear my gown and take pictures with students, and my colleagues announce the names of the graduating cohort, their achievements, and where they are going. I am always amazed by the amount of achievement and the bright futures these kids have; they bring me joy. But this year, although nobody mentions AI, it is always on top of my head.

I have been playing squash at the Stanford squash court with two partners. One is a cardiothoracic surgeon and the other is a tech lead. Last week, one of them asked me, “Yiqing, is there anything wrong? I feel that in the past few months, you seem different.”

I confessed to him that I have felt disarrayed, and I attribute this to AI.

Don’t get me wrong: AI has brought huge excitement to our field and to me personally. I have been able to delegate a lot of the more tedious work to AI, such as writing software packages and replicating existing research.

Almost no one writes (a significant amout of) code by hand anymore; that is simply the reality now. For fect, one of my R packages, AI has written hundreds of tests and caught many bugs along the way, some of which had gone unnoticed for years—see its changelog; the treatment effect of AI is mind-blowing. I am a huge beneficiary of the agentic AI that arrived this year.

What I am certain

Now that we have been in this agentic world for a few months, I am quite certain that AI will fundamentally change the entire business of social science, and more broadly, science.

This might have been a provocative statement a few months ago. It feels much less so now. As I have said on different occasions, the tsunami has already come, and we have to adapt whether we like it or not.

I think the quality of social science papers will increase dramatically—maybe not even in two years, but perhaps in just a few months. This is especially true for quantitative and statistical papers, which I am more familiar with, simply because of the cognitive (maths and programming) help we get from AI.

I still see people on Twitter from time to time questioning AI’s capabilities, usually by pointing to hallucinations or its inability to get things right in one shot. But I think these issues, once clearly identified, will become obselete very quickly. Even compared with two months ago, they are much rarer now.

I’m not saying that we don’t need to validate. In fact, we need to validate a ton, as I will argue later.

But it has also brought a deep sense of questioning, uncertainty, and anxiety that I try to describe but don’t quite know how to.

Sources of anxiety

I’m trying to reflect on why I feel disarrayed recenlty, despite the fact that I keep producing work with the help of AI. Writing this blog is in fact a meditation exercise for myself.

More than a productivity tool

The first reason is that AI is not only a productivity tool. What it enhances or replaces is the cognitive ability we see as central to who we are, so it starts to touch the meaning of work and life. This is not a new worry. People in the humanities and the arts have raised it for a long time, often to skepticism from people like me. I feel it now myself.

Gary used to tell us, and I paraphrase, that ideas are cheap but execution is king (excuse me for the pun). However, with AI, the cost of execution is becoming very low. Does this mean the idea becomes king again?

I am not sure. I was debating this with my former student (who works in an AI startup now) today, and he tried to convince me that what I perceive as cheap ideas might actually be good ones, worth sharing with the world at a faster pace. But when the actual work is hollowed out by AI, are those ideas actually human? Do they truly represent human ingenuity?

Maybe. But I feel anxious because we are in this transition period where a lot of low-hanging fruit ideas are still there. Perhaps after this transition, what remains will be much more difficult problems, where I can again find a deeper meaning in the work.

A friend of mine—a very smart guy whom I admire a great deal—told me that he feels we are all astronomers now. No offense to astronomers, because neither he nor I know exactly what your job is, but his point was this: we are trying to find stars using a telescope that wasn’t built by us. We are just staring at it, sometimes steering it a bit and trying to make a discovery, oftentimes by luck.

Information overload

The second reason I reflect is the overwhelming amount of information and choices we face.

When we are presented with too much information and too many choices, it creates a situation with several different layers. On the one hand, marginal productivity becomes exceptionally high for individual researchers who have a good command of their field and their work.

On the other hand, you really have to choose what to work on. Even before AI, this was a choice and a salient problem for many researchers; we often had to balance the quantity-quality trade-off.

Now, however, it has become such a pressing problem because reasonably good work can be produced much faster than before. If we are able to produce 50 times more work that is equally as good as the work we deem “good” in 2025, should we do it? Or should we wait and instead tackle the really difficult problems? I think the professional incentive will favor the former, especially for graduate students and pre-tenure faculty, rather than the latter.

Then there is understanding. Someone famous (probably Terence Tao, but I cannot find the exact quote) said that we can outsource thinking, but we cannot outsource understanding. I feel this very strongly these days because what AI produces is just overwhelming.

The problem is that most of it is right—it is good information. Only about 5-10% (in areas I interacted with, and it’s shrinking) is highly problematic, and my job is to figure out what that 5-10% is and steer the AI to correct it. Is that the nature of our job? I don’t think it will be eventually, but that is the current status of my work. Every day I feel my brain is being fried, and I use up my cognitive tokens fairly quickly.

I hear people in academia often say, “Oh no, we need to read every line of the code that the machine generates and make sure it’s right.” But do we? Do we really do that? I don’t think that would even be feasible.

We need much smarter ways to validate—in software development, for example, this means using information barriers, adversarial tests, and workflows in which one component checks another.

Signal jamming

The last source of anxiety, which is not a deep personal one but perhaps more relevant for my students, is that the signals of ability are being distorted in a dramatic way.

In the past, for example, if you could prove semiparametric efficiency bounds, that signaled your ability as an econometrician. I heard that signal is much weaker now: with a few good prompts, a frontier model will often hand you something pretty much correct out of the gate, even if you still have to check it carefully.

This raises a few critical questions:

1. How are we going to select a good scholar? We can always go back to the standard answer that we want scholars to work on ambitious, big, and deep questions, but I imagine the transition period will be chaotic.

2. How do we guide skill accumulation for students (and even for ourselves) in this highly dynamic environment? The downstream consequence is that we simply don’t know which skills will be more cherished, helpful, or useful in six months.

---

I have left out many of the optimistic views on the role of humans in social science discovery in the age of AI. For example, societies are notoriously slow to change, data are hard for AI to collect directly, a lot more questions will become open, and there is a long tail of social discovery that AI is still not very good at. I don’t disagree with them. But since others have already made these points, I will not repeat them here.

Path ahead

It is very dangerous and imprudent to make any predictions in this day and age. However, given what I know and my experience with AI, I believe a few things are certain:

1. A lot more emphasis on validation. We are now all AI trainers—the most direct way is that we filter out errors generated by AI and submit our papers to open-source platforms like arXiv. Whether we like it or now, one of our key roles is to remove incorrectness from the knowledge pipeline so that AI-assisted work remains trustworthy. As Matt Gentzkow said in the April event, trust will be an increasingly scarce commodity. If trust is broken, scholarly reputation will be hurt a great deal.

2. The need to slow down. I know this is a prisoner’s dilemma in many situations. Even if we assume away the external incentives, the internal intrinsic incentive to create and produce more is still very tempting because AI is such a godsend toy for those of us who want to discover and make things happen.

I have another point to add. I think we should be more tolerant of students’ use of AI, especially since they are still in a kind of apprenticeship. As long as they are acting in good faith, the goal should not be to police every use, but to help them understand the risks and limitations.

The misuse of AI will be common in the coming months and years. I think there is a tendency to over-moralize it, to treat every case as breaking a moral code. The norms are being rewritten month by month, and most students are working with good intentions, using tools that shifted under their feet, under a lot of pressure.

The real lines should still hold: fabrication, or passing off work you don’t understand as your own, is wrong, and we should say so—I acknoweldge the temptation is strong. Short of that, we should give students room to experiment, while warning them, repeatedly, about how things can go wrong.

If I were to give any advice to grad students, I would make three additional points:

1. Integrate AI as much as possible into your workflow. I am sure most of you are doing this already, so it is probably moot.

2. Try to work on much more ambitious projects than what you are seeing now in job talks.

3. Focus a lot more on understanding and self-teaching. I am quite certain that a deeper understanding of a problem is going to amplify the capability of AI, and vice versa. Your work, when assisted by AI, will be much better if you truly understand the underlying issues. I have seen this with my own students.

I apologize for striking this little bit of a melancholy tone during this festive season.