Measurement in the Age of AI: Discovery, Definition, and Observation

Melissa Dell (Harvard), part 1 of 4 of the NBER Summer Institute 2026 Methods Lecture “Estimation and Inference with AI-Generated Data” (with Ashesh Rambachan), July 30, 2026. Slides: dell4.pdf; companion background reading: Dell’s “Deep Learning for Economists”. Timestamps refer to the lecture video.

The methods lecture is NBER’s annual institution for telling the profession what it’s about to do wrong, and this year’s subject is the thing half the audience did last week: feeding text into a language model and putting the output on the right-hand side of a regression. Dell’s opening move is to refuse to start with the AI at all. She starts with a question that sounds insultingly basic — what is measurement? — on the grounds that most disagreements about AI-generated data turn out to be disagreements about this.

Her answer: measurement is representation. It maps a high-dimensional reality into low-dimensional variables, deliberately preserving some features and discarding the rest — “we wouldn’t plug a book directly into our regression, but we might measure something about that book.” The discarding is not a bug; it’s the entire function. She reaches for Borges’s cartographers, who perfect their craft until the map is the size of the empire and thereby useless: a representation that simplifies nothing has ceased to be a map. So measurement is “a disciplined, fallible, purpose-oriented attempt to learn from reality through simplified representations,” and since there are many ways to project the same reality down to a variable, different projections can support different empirical conclusions. The lecture series is organized around the three stages where those choices get made: discovery (what exists and is worth measuring?), construct definition (how should the concept be operationalized?), and observation (how do you implement the definition at scale, and how do the errors propagate?). Each stage has its own discipline — discovery traditionally belongs to theory and exploratory analysis, construct definition to psychometrics, observation to econometrics — and AI is barging into all three at once.

The economic history framing is the lecture’s best analytical move. Traditionally, measurement was expensive, so it was treated as fixed: you inherited whichever proxy existed (expropriation risk standing in for property rights), justified it, and moved on. Nobody compared measurement functions because nobody could afford two. Neural networks change the cost structure — sometimes. Dell is careful here: for her own historical-documents work, credible AI measurement still demands heavy fixed investments in training data and fine-tuning. But where off-the-shelf models work, generating many candidate measures of the same concept becomes nearly free, and the bottleneck shifts from building any measure to choosing among them. Her analogy is exact and pointed: this is the 1990s all over again, when cheap computation turned one regression into two million and forced the field to invent the credibility revolution — causal inference, robustness, multiple testing — to discipline specification search. “AI moves specification search upstream,” from choosing among regressions to choosing among representations of reality. Without a principled response, she warns, the field gets both failure modes at once: “p-hacking with AI slop on the one hand, and nihilism about what can be learned on the other. This is already happening.”

Her demonstration is uncomfortably easy to replicate. Take economic policy uncertainty — the Baker–Bloom–Davis concept — and ask frontier LLMs (she used GPT and Opus) to write the measurement rubric, ten times. Each rubric sounds plausible. Implemented, they correlate positively but far from perfectly:

Ten defensible-sounding measures of the same concept, each potentially supporting different downstream conclusions, produced in an afternoon. “We have no idea really if this is slop or if this is capturing meaningful variation.”

The proposed discipline is validation, and Dell spends the lecture’s second half armoring it against every objection she’s collected. Validation cannot show a measure captures latent truth — the goal is more modest: state an explicit criterion and assess the measure against it, because an explicit criterion can be interpreted, audited, debated, and refined, which is precisely what a neural network’s internal decision rule cannot be. (Validation, she insists, doesn’t privilege humans — the criterion data can come from instruments, admin records, or theory — it privileges interpretable criteria over implicit rules learned by a black box.) Each measurement stage gets its own validation logic. For discovery: internal coherence and stability under perturbation, plus a 1959 vintage import from psychometrics — Campbell and Fiske’s multitrait-multimethod framework — reincarnated as “vary the embedding model, the clustering algorithm, the data source, and see if the structure survives,” plus sample-splitting to keep post-selection inference honest (her practical gem: you can pre-register a discovery procedure for open-ended survey responses without pre-specifying what you’ll find). For construct definition: Cronbach and Meehl’s nomological network — a construct is validated by the whole pattern of what it should correlate with (convergent validity) and stay distinct from (discriminant validity) — which Dell recasts, for the econometricians in the room, as “partial identification under theoretically motivated moment restrictions.” Economists never needed this while they measured prices and schooling; now that they measure trust and uncertainty from text, they’ve inherited psychology’s hardest problem without its fifty years of tooling.

The lecture’s sharpest empirical teeth come at the observation stage, aimed at the profession’s favorite comfort blanket: “I ran it through several AI models and they agreed.” All frontier models share the transformer architecture, similar training procedures, a snapshot of the same internet, and RLHF pushing toward similar preferred answers — so agreement can reflect shared bias, not quality, and one model’s output cannot instrument for another’s (the errors are correlated, violating the exclusion restriction). Dell shows this with a hand-labeled ground-truth corpus of historical newspaper articles classified for “politics”: prediction errors are highly correlated across models, and:

Every model overshoots, because models fall back on pre-trained notions of “politics” (a 1967 measles-clinic announcement isn’t political under the rubric, however political measles is on today’s internet). This is the general point about AI measurement error: it is produced by nonlinear transformations, trained-in priors and classification thresholds, so there is no reason to expect it to be classical, and biased predictions propagate straight into the target parameter. The fix — using a small, costly, criterion-based validation sample to debias the huge, cheap, black-box-measured sample, so that any model, good or bad, yields the same (merely noisier) estimate — is the subject of her second lecture, and she exits on that cliffhanger.

What the lecture really is, underneath, is a property claim on behalf of explicitness. The map is not the territory, and nobody ever thought it was; the danger of AI is not that it makes maps, but that it makes them by the thousand, beautifully, with the cartographic conventions hidden inside a billion parameters. Dell’s whole program is that you may use the black-box cartographer — you just have to check its maps against a landmark you can name.