Discovery and Hypothesis Generation with Unstructured Data

Ashesh Rambachan (MIT), part 2 of 4 of the NBER Summer Institute 2026 Methods Lecture “Estimation and Inference with AI-Generated Data” (with Melissa Dell), July 30, 2026. Slides: ash1.pdf; the companion background reading: Ludwig & Mullainathan’s “Machine Learning as a Tool for Hypothesis Generation”. Timestamps refer to the lecture video.

Economists have a standing joke about the drunk looking for his keys under the lamppost, and Rambachan’s lecture is what happens when someone takes the joke as a research agenda. Structured data — the closed survey items, the administrative extract’s fields, the outcomes you pre-registered — is the lamppost. The interesting features of the world are mostly out in the dark, and the reason we don’t measure them isn’t that they’re unmeasurable but that we cannot list them in advance. That’s Polanyi’s paradox, restated for empirical economics: we can recognize a good hypothesis when we see one, but we can’t enumerate the space of them.

The opening motivation is three observations the machine-learning-meets-economics literature has accumulated over a decade. First, algorithms routinely out-predict our best theories out of sample, even in classic settings like decision-making under risk — which raises the question of what signal they found that we missed. Second, prediction isn’t what economists want; we want mechanisms, counterfactuals, portability, and a black box supplies none of them. Third, the lamppost. Put together, they suggest a specific opportunity: use AI on unstructured data to automatically discover what to measure and how it relates to outcomes — the activity this literature calls hypothesis generation — and then feed the discovered features into the ordinary empirical toolkit.

Three running examples anchor the talk. In pretrial release, Ludwig and Mullainathan turned the camera around: instead of predicting defendant misconduct, predict the judge’s detention decision — and found the single strongest predictor is the mug shot, with about 78% of the mug shot’s predictive signal unexplained by charge, priors, demographics, known psychological facial features, or even incentivized human guesses. Something unnamed in the face is driving detention. In child protective services, Rambachan’s work with Baron, Dobbie, Lombardi and Ryan found the opposite ranking — roughly 36% of Michigan investigators significantly out-perform the algorithm — with the gap unexplained by structured data; but investigators write rich case notes documenting home visits and interviews. And in macro, Andre et al. collected open-ended explanations of the 2021–22 inflation surge, then hand-coded every response into a directed acyclic graph of narratives, which is exactly the painstaking human labor a machine might now do.

The lecture’s organizing contribution is a common architecture, which turns a zoo of papers across economics, CS, medicine and computational social science into one recipe with four slots:

Scoring attaches a number to each unstructured object: the raw outcome, a learned predictor of the outcome, or an embedding. Rambachan argues against the raw outcome — a fitted predictor keeps only the variation in $y$ the unstructured data can explain, and it can score units that never appeared in the data, which matters downstream. Contrasting builds sets of examples that differ along the score, in one of three ways: morphs (generate a counterfactual pair by moving along the score’s gradient), tails (take real high- and low-scoring units), or activations (take units that fire on one direction of a sparse autoencoder’s embedding space). Interpretation hands the contrast set to someone — humans on Prolific, domain experts, or an LLM — to name the difference. Evaluation takes the named features to a held-out test set and asks whether they predict the outcome.

The morphing example is the lecture’s best set piece, because the naive version fails so instructively. Take a mug shot, follow the detention predictor’s gradient in pixel space, and you produce, in Rambachan’s words, “something decidedly not a face” — “a little more haunting than a photo of a face.” The diagnosis is geometric: real faces occupy a vanishingly low-dimensional manifold in the space of all pixel arrays, so almost any direction steps off it. The fix is to morph inside the latent space of a generative model (a GAN) of mug shots and decode. Sanity-checked against a known feature — morph along an age predictor, and subjects reliably see aging — the procedure is then pointed at the detention predictor:

Then the elegant iteration: having named well-groomedness, morph again in a direction orthogonal to it, and a second hypothesis appears — heavy-faced — also strongly predictive out of sample, and also, per the paper’s extensive novelty checks, new to the psychology literature, the crime literature, and to practicing prosecutors and judges. The two features plus every known psychological facial characteristic explain about 27% of the CNN’s variation, which is Rambachan’s way of saying there is plenty of undiscovered structure left in a defendant’s face. Nothing here is about faces: the same pipeline, with a generative model of ECG waveforms and a sudden-cardiac-death predictor, produced a novel morphological ECG feature in work by Obermeyer and coauthors.

Text resists morphing — steering an LLM’s latent space and decoding something sensible is, Rambachan notes with visible restraint, “a very non-trivial problem” he’ll bracket — so the case-notes paper uses tails instead: bin cases by predicted opening probability from structured data, then within a bin contrast notes that the text predicts will open against those it predicts won’t, so the contrast isolates the text. An LLM reads and names five recurring evidence themes, which turn out to explain 84% of the case notes’ predictive signal for case-opening on held-out data — and, the substantive payoff, high-performing investigators are systematically more likely to record disconfirming evidence: ambiguity, contradiction of the initial referral, corroboration that care is adequate. That is textbook confirmation-bias correction, discovered rather than hypothesized, and it converts directly into a trainable intervention.

For the inflation narratives, Rambachan reports doing the reanalysis “in one day using Codex”: off-the-shelf OpenAI embeddings of the open-ended responses, a sparse autoencoder (the HypothesAEs recipe from CS) to unbundle the embedding’s polysemantic directions into sparse interpretable ones, train/validation/test splits to pick the twenty directions most correlated with inflation expectations, then an LLM to name them. Many recovered concepts line up with the authors’ hand-coding; some don’t — including, memorably, whether the respondent refers to Joe Biden by nickname, which is very predictive of inflation expectations and which you might gloss as hyperpartisanship.

The methodological warnings are pointed and, for once, quantified. Interpreters are not interchangeable: humans and LLMs carry different inductive biases and will name different things from identical contrast sets. Multiplicity is severe — one CS pipeline generated over 3,000 candidate hypotheses, of which about 13 survived multiple-testing correction — and since researchers then pick the most interesting survivors for downstream work, the selective-inference and inference-on-winners machinery applies directly. And completeness — how much of the original unstructured signal the named features retain, benchmarked against a flexible predictor using the raw data — is the discipline that stops a tidy story from quietly discarding most of the variation. His open methodological question is whether validity and completeness are properties of the hypotheses or of the procedure, and whether there’s a size–completeness tradeoff to be characterized: an econometrics of discovery that mostly doesn’t exist yet.

He closes with a recruiting pitch disguised as a summary: the barrier to entry is low (hand an LLM contrasting documents, ask what differs, validate out of sample; the fancier SAE version ships as a package), the frontier moves monthly, and the scarce input is not compute or code but your comparative advantage — knowing which economic setting has interesting unstructured data tied to an outcome anyone cares about. Then the handoff to the second half: a generated hypothesis is only a named concept. Somebody still has to measure it, at scale, without the measurement error eating the estimate — which is where Dell picks up after the break.