Measurement with Validation Samples: A Missing-Data Perspective

Melissa Dell (Harvard), part 3 of 4 of the NBER Summer Institute 2026 Methods Lecture “Estimation and Inference with AI-Generated Data” (with Ashesh Rambachan), July 30, 2026. Slides: dell3.pdf (construct validity) and dell6.pdf (observation); the underlying paper is Carlson & Dell, “A Unifying Framework for Robust and Efficient Inference with Unstructured Data”. Timestamps refer to the lecture video.

Dell opens by naming the reason most of the room showed up: “the main reason people might choose to view the lectures is they want to know, how should I compute my standard errors? How do I run a differences-in-differences specification with AI-predicted variables?” She promises to get there, and to hand out an R package. But first, ten minutes on the problem she has no package for.

Construct validity as partial identification. In lecture one she showed that ten LLM-written rubrics for “economic policy uncertainty” correlate only 0.5–0.8 with each other. Here she sketches the econometric response: treat Cronbach and Meehl’s nomological network as a set of moment restrictions and the choice of measure as a partial identification problem. The researcher specifies a menu of candidate measurement functions mapping the same high-dimensional reality into comparable measures, plus auxiliary observables that theory says the construct should correlate with (convergent validity), stay distinct from (discriminant validity), or differ across (a measure of partisan speech had better separate Republicans from Democrats). Those become population moment inequalities; the measures satisfying all of them are the admissible set; running each admissible measure through the downstream regression traces out a construct-identified set for the parameter you actually care about. Two sources of uncertainty then have to be combined — sampling error in each $\hat\beta$, and sampling error in the moment inequalities that decide admissibility — which existing partial-identification and endpoint-inference machinery can handle, conservatively. She reports slack in each restriction so you can see which restriction is doing the work, and flags a lovely second-order worry she has nothing to say about: construct validity is use-contingent. A measure validated as passive description can lose validity once it’s embedded in decisions and agents re-optimize — Goodhart’s law for construct definition, made worse by AI’s ability to deploy constructs at scale in real time. Her honest summary: this is wide open, there’s no package, and “with AI we just can’t avoid this question.”

Then the well-developed half. Observation assumes the construct is given and asks whether the scalable procedure implements it faithfully. Treating AI output as a proxy — the historical default, since hand-coding was expensive — creates four specific problems in a world of cheap LLMs: different models and implementation details produce different predictions and hence post-selection concerns; sensitivity checks can’t rescue you because biases are correlated across models; proprietary models raise reproducibility problems (you used Opus for the paper, it’s deprecated by publication, the successor gives a different answer); and there’s no principled way to judge whether costly investments in better models are worth it.

The reframing that unlocks everything: inference with AI predictions is a missing-data problem. You have a hundred million newspaper articles; what’s missing is the low-dimensional label — does this article discuss economic policy uncertainty? — that your estimating equation needs. Once you say it that way, fifty years of Rubin-style missing-data theory applies, and the assumptions line up one-for-one with causal inference: consistency of potential outcomes; missing at random (conditional on observables, annotated and unannotated data are comparable in their ground-truth labels — “you could call this annotation on observables”); and a known, bounded annotation score function π (overlap). Ground truth here means the machine-learning sense: the label your stated rubric would assign, not fundamental truth. The framework is MAR-S (missing at random structured data), joint with Jacob Carlson.

The crucial point about why this framing suits neural networks: MAR requires almost no assumptions about the model. Network biases shift in complicated, unpredictable ways as inputs shift — a crop classifier trained in the US will fail in South Sudan in ways nobody can forecast — so any framework that needs you to model the error structure is doomed. MAR sidesteps it entirely. And to the objection that overlap is implausible when you can label only a vanishing fraction of a billion documents, she points at the causal-inference literature on decaying overlap: nothing breaks if the number of annotations diverges while the labeled share goes to zero.

The workhorse estimator is AIPW. For a mean, following Chen et al. (2008):

where $M_i$ is the ground-truth label observed when the annotation indicator $A_i=1$, $\hat\mu$ is the AI’s imputation, and $\pi$ is the annotation probability (slide 25 of dell6). Dell’s plain-language decomposition is the part to remember: the second term is what you’d report if you ignored that the data were AI-generated; the first is an estimate of the model’s measurement error in the annotated sample, reweighted. A perfect model zeroes out the correction; a terrible model produces a large, high-variance correction that pulls the estimate back toward the annotated data and blows up the standard errors — which is exactly the behavior you want, since it prices your measurement quality into your inference. Everything else in the lecture is this same move applied to OLS with a missing outcome or regressor, IV with a missing outcome or instrument, and differences in conditional means (which covers diff-in-diff, RD, and RCT treatment effects). Two properties fall out: a double-robustness flavor — any imputation function gives valid inference, even a stupid one, just with enormous intervals — and the efficiency result that you should still make the model as accurate as you can.

The demonstration uses Baker, Bloom and Davis’s economic policy uncertainty index, chosen precisely because it’s among the most careful and most cited text-analysis papers in economics and has an unusually extensive human audit sample:

The debiased series shows modest downward bias and much wider confidence intervals, with the keyword classifier penalized more than the neural one — exactly the ranking you’d want a validation framework to produce.

Aggregation is the piece Dell says motivated the whole project, because it’s what applied economists actually do and what the existing literature doesn’t cover: you predict at the article level, average up to a country-year, take a log, interact it with firm exposure, and run the regression — but the debiasing literature assumes ground truth at the level of the estimating equation. The fix is pleasingly simple. Debias the aggregate with the MAR-S first-step estimator (you do have validation data at the article level), then plug it in. Debiasing doesn’t eliminate measurement error; it converts systematic error into classical error — at which point the standard corrections, familiar for decades and available in Stata, extend naturally to clustering, panels, and heteroskedastic variance. (Her defense of the usual complaint about knowing the measurement-error variance: you know it precisely when the sample is huge, which is the situation you’re in.)

The conclusions survive here — as Dell says, this is “not the paper I’m worried about.” The papers she is worried about are the ones with no validation data at all.

The practical coda answers the question every attendee had: how many labels? Often, for a binary indicator, “a few hundred is enough” — not prohibitive, and, as she notes with a straight face, all the time you’re saving by having coding agents run your regressions should cover it. More annotations help; a more accurate imputation model helps more, at least for the standard errors. And the framework’s boundary is the assumption at its center: what happens when you can’t credibly claim missing-at-random — when there is no random validation sample to be had? That’s Rambachan’s closing lecture.